经典分词器
编辑经典分词器
编辑classic 分词器是一个基于语法的分词器，适用于英语文档。此分词器对缩写、公司名称、电子邮件地址和互联网主机名有特殊的处理启发式方法。然而，这些规则并不总是有效，而且该分词器对于除英语以外的大多数语言效果不佳。
- 它在大多数标点符号处分割单词，并移除标点符号。但是，如果一个点后面没有空格，则该点被视为标记的一部分。
- 它在连字符处分割单词，除非标记中包含数字，在这种情况下，整个标记被解释为产品编号，并且不分割。
- 它将电子邮件地址和互联网主机名识别为一个标记。
示例输出
编辑resp = client.indices.analyze(
tokenizer="classic",
text="The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.",
)
print(resp)
response = client.indices.analyze(
body: {
tokenizer: 'classic',
text: "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
)
puts response
const response = await client.indices.analyze({
tokenizer: "classic",
text: "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.",
});
console.log(response);
POST _analyze
{
"tokenizer": "classic",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
上述句子将产生以下词条
[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone ]
配置
编辑classic 分词器接受以下参数：
|
|
最大标记长度。如果发现标记超过此长度，则将在 |
示例配置
编辑在此示例中，我们将 classic 分词器配置为具有 5 的 max_token_length （用于演示目的）
resp = client.indices.create(
index="my-index-000001",
settings={
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "classic",
"max_token_length": 5
}
}
}
},
)
print(resp)
resp1 = client.indices.analyze(
index="my-index-000001",
analyzer="my_analyzer",
text="The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.",
)
print(resp1)
response = client.indices.create(
index: 'my-index-000001',
body: {
settings: {
analysis: {
analyzer: {
my_analyzer: {
tokenizer: 'my_tokenizer'
}
},
tokenizer: {
my_tokenizer: {
type: 'classic',
max_token_length: 5
}
}
}
}
}
)
puts response
response = client.indices.analyze(
index: 'my-index-000001',
body: {
analyzer: 'my_analyzer',
text: "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
)
puts response
const response = await client.indices.create({
index: "my-index-000001",
settings: {
analysis: {
analyzer: {
my_analyzer: {
tokenizer: "my_tokenizer",
},
},
tokenizer: {
my_tokenizer: {
type: "classic",
max_token_length: 5,
},
},
},
},
});
console.log(response);
const response1 = await client.indices.analyze({
index: "my-index-000001",
analyzer: "my_analyzer",
text: "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.",
});
console.log(response1);
PUT my-index-000001
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "classic",
"max_token_length": 5
}
}
}
}
}
POST my-index-000001/_analyze
{
"analyzer": "my_analyzer",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
上面的示例产生以下词条
[ The, 2, QUICK, Brown, Foxes, jumpe, d, over, the, lazy, dog's, bone ]