标准分词器
编辑标准分词器编辑
standard 分词器提供基于语法的分词（基于 Unicode 文本分段算法，如 Unicode 标准附件 #29 中所述），适用于大多数语言。
示例输出编辑
response = client.indices.analyze(
body: {
tokenizer: 'standard',
text: "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
)
puts response
POST _analyze
{
"tokenizer": "standard",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
以上句子将产生以下词语
[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone ]
配置编辑
standard 分词器接受以下参数
|
|
最大词语长度。如果遇到超过此长度的词语，则将其在 |
示例配置编辑
在此示例中，我们将 standard 分词器的 max_token_length 配置为 5（为了演示目的）
response = client.indices.create(
index: 'my-index-000001',
body: {
settings: {
analysis: {
analyzer: {
my_analyzer: {
tokenizer: 'my_tokenizer'
}
},
tokenizer: {
my_tokenizer: {
type: 'standard',
max_token_length: 5
}
}
}
}
}
)
puts response
response = client.indices.analyze(
index: 'my-index-000001',
body: {
analyzer: 'my_analyzer',
text: "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
)
puts response
PUT my-index-000001
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "standard",
"max_token_length": 5
}
}
}
}
}
POST my-index-000001/_analyze
{
"analyzer": "my_analyzer",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
以上示例产生以下词语
[ The, 2, QUICK, Brown, Foxes, jumpe, d, over, the, lazy, dog's, bone ]