字符组分词器
编辑字符组分词器
编辑char_group 分词器会在遇到定义的字符集中的字符时将文本分割成词项。它主要适用于需要简单自定义分词且不能接受 pattern 分词器开销的情况。
配置
编辑char_group 分词器接受一个参数：
|
|
一个列表，包含用于分割字符串的字符列表。每当遇到此列表中的字符时，就会开始一个新的词项。它接受单个字符，例如 |
|
|
最大词项长度。如果看到的词项超过此长度，则会在 |
示例输出
编辑resp = client.indices.analyze(
tokenizer={
"type": "char_group",
"tokenize_on_chars": [
"whitespace",
"-",
"\n"
]
},
text="The QUICK brown-fox",
)
print(resp)
response = client.indices.analyze(
body: {
tokenizer: {
type: 'char_group',
tokenize_on_chars: [
'whitespace',
'-',
"\n"
]
},
text: 'The QUICK brown-fox'
}
)
puts response
const response = await client.indices.analyze({
tokenizer: {
type: "char_group",
tokenize_on_chars: ["whitespace", "-", "\n"],
},
text: "The QUICK brown-fox",
});
console.log(response);
POST _analyze
{
"tokenizer": {
"type": "char_group",
"tokenize_on_chars": [
"whitespace",
"-",
"\n"
]
},
"text": "The QUICK brown-fox"
}
返回
{
"tokens": [
{
"token": "The",
"start_offset": 0,
"end_offset": 3,
"type": "word",
"position": 0
},
{
"token": "QUICK",
"start_offset": 4,
"end_offset": 9,
"type": "word",
"position": 1
},
{
"token": "brown",
"start_offset": 10,
"end_offset": 15,
"type": "word",
"position": 2
},
{
"token": "fox",
"start_offset": 16,
"end_offset": 19,
"type": "word",
"position": 3
}
]
}