EQL 搜索
编辑EQL 搜索
编辑事件查询语言 (EQL) 是一种用于基于事件的时间序列数据(例如日志、指标和跟踪)的查询语言。
EQL 的优点
编辑-
EQL 可以让您表达事件之间的关系。
许多查询语言允许您匹配单个事件。EQL 可以让您匹配不同事件类别和时间跨度内的一系列事件。 -
EQL 的学习曲线较低。
EQL 语法 看起来与其他常见的查询语言(例如 SQL)相似。EQL 使您可以直观地编写和读取查询,从而实现快速、迭代的搜索。 -
EQL 专为安全用例而设计。
虽然您可以将其用于任何基于事件的数据,但我们创建 EQL 是为了进行威胁狩猎。EQL 不仅支持入侵指标 (IOC) 搜索,还可以描述超出 IOC 范围的活动。
必需字段
编辑除了示例查询外,EQL 搜索要求搜索的数据流或索引包含一个 *timestamp* 字段。默认情况下,EQL 使用来自 Elastic Common Schema (ECS) 的 @timestamp
字段。
EQL 搜索还需要一个 *event category* 字段,除非您使用 any
关键字 来搜索没有事件类别字段的文档。默认情况下,EQL 使用 ECS event.category
字段。
要使用不同的时间戳或事件类别字段,请参阅 指定时间戳或事件类别字段。
虽然使用 EQL 不需要任何架构,但我们建议使用 ECS。EQL 搜索默认设计为与核心 ECS 字段一起使用。
运行 EQL 搜索
编辑使用 EQL 搜索 API 来运行 基本的 EQL 查询。
resp = client.eql.search( index="my-data-stream", query="\n process where process.name == \"regsvr32.exe\"\n ", ) print(resp)
response = client.eql.search( index: 'my-data-stream', body: { query: "\n process where process.name == \"regsvr32.exe\"\n " } ) puts response
const response = await client.eql.search({ index: "my-data-stream", query: '\n process where process.name == "regsvr32.exe"\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "query": """ process where process.name == "regsvr32.exe" """ }
默认情况下,基本 EQL 查询在 hits.events
属性中返回 10 个最近匹配的事件。这些命中按时间戳排序,转换为自 Unix 纪元以来的毫秒数,并按升序排列。
{ "is_partial": false, "is_running": false, "took": 60, "timed_out": false, "hits": { "total": { "value": 2, "relation": "eq" }, "events": [ { "_index": ".ds-my-data-stream-2099.12.07-000001", "_id": "OQmfCaduce8zoHT93o4H", "_source": { "@timestamp": "2099-12-07T11:07:09.000Z", "event": { "category": "process", "id": "aR3NWVOs", "sequence": 4 }, "process": { "pid": 2012, "name": "regsvr32.exe", "command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll", "executable": "C:\\Windows\\System32\\regsvr32.exe" } } }, { "_index": ".ds-my-data-stream-2099.12.07-000001", "_id": "xLkCaj4EujzdNSxfYLbO", "_source": { "@timestamp": "2099-12-07T11:07:10.000Z", "event": { "category": "process", "id": "GTSmSqgz0U", "sequence": 6, "type": "termination" }, "process": { "pid": 2012, "name": "regsvr32.exe", "executable": "C:\\Windows\\System32\\regsvr32.exe" } } } ] } }
使用 size
参数来获取较小或较大的命中集
resp = client.eql.search( index="my-data-stream", query="\n process where process.name == \"regsvr32.exe\"\n ", size=50, ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", query: '\n process where process.name == "regsvr32.exe"\n ', size: 50, }); console.log(response);
GET /my-data-stream/_eql/search { "query": """ process where process.name == "regsvr32.exe" """, "size": 50 }
搜索一系列事件
编辑使用 EQL 的 序列语法 来搜索一系列按顺序排列的事件。按时间顺序升序列出事件项,最近的事件列在最后
resp = client.eql.search( index="my-data-stream", query="\n sequence\n [ process where process.name == \"regsvr32.exe\" ]\n [ file where stringContains(file.name, \"scrobj.dll\") ]\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", query: '\n sequence\n [ process where process.name == "regsvr32.exe" ]\n [ file where stringContains(file.name, "scrobj.dll") ]\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "query": """ sequence [ process where process.name == "regsvr32.exe" ] [ file where stringContains(file.name, "scrobj.dll") ] """ }
响应的 hits.sequences
属性包含 10 个最近匹配的序列。
{ ... "hits": { "total": ..., "sequences": [ { "events": [ { "_index": ".ds-my-data-stream-2099.12.07-000001", "_id": "OQmfCaduce8zoHT93o4H", "_source": { "@timestamp": "2099-12-07T11:07:09.000Z", "event": { "category": "process", "id": "aR3NWVOs", "sequence": 4 }, "process": { "pid": 2012, "name": "regsvr32.exe", "command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll", "executable": "C:\\Windows\\System32\\regsvr32.exe" } } }, { "_index": ".ds-my-data-stream-2099.12.07-000001", "_id": "yDwnGIJouOYGBzP0ZE9n", "_source": { "@timestamp": "2099-12-07T11:07:10.000Z", "event": { "category": "file", "id": "tZ1NWVOs", "sequence": 5 }, "process": { "pid": 2012, "name": "regsvr32.exe", "executable": "C:\\Windows\\System32\\regsvr32.exe" }, "file": { "path": "C:\\Windows\\System32\\scrobj.dll", "name": "scrobj.dll" } } } ] } ] } }
使用 with maxspan
将匹配序列限制在时间跨度内
resp = client.eql.search( index="my-data-stream", query="\n sequence with maxspan=1h\n [ process where process.name == \"regsvr32.exe\" ]\n [ file where stringContains(file.name, \"scrobj.dll\") ]\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", query: '\n sequence with maxspan=1h\n [ process where process.name == "regsvr32.exe" ]\n [ file where stringContains(file.name, "scrobj.dll") ]\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "query": """ sequence with maxspan=1h [ process where process.name == "regsvr32.exe" ] [ file where stringContains(file.name, "scrobj.dll") ] """ }
使用 !
来匹配缺失的事件:序列中在给定时间跨度内不满足条件的事件
resp = client.eql.search( index="my-data-stream", query="\n sequence with maxspan=1d\n [ process where process.name == \"cmd.exe\" ]\n ![ process where stringContains(process.command_line, \"ocx\") ]\n [ file where stringContains(file.name, \"scrobj.dll\") ]\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", query: '\n sequence with maxspan=1d\n [ process where process.name == "cmd.exe" ]\n ![ process where stringContains(process.command_line, "ocx") ]\n [ file where stringContains(file.name, "scrobj.dll") ]\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "query": """ sequence with maxspan=1d [ process where process.name == "cmd.exe" ] ![ process where stringContains(process.command_line, "ocx") ] [ file where stringContains(file.name, "scrobj.dll") ] """ }
缺失的事件在响应中指示为 missing": true
{ ... "hits": { "total": ..., "sequences": [ { "events": [ { "_index": ".ds-my-data-stream-2023.07.04-000001", "_id": "AnpTIYkBrVQ2QEgsWg94", "_source": { "@timestamp": "2099-12-07T11:06:07.000Z", "event": { "category": "process", "id": "cMyt5SZ2", "sequence": 3 }, "process": { "pid": 2012, "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } }, { "_index": "", "_id": "", "_source": {}, "missing": true }, { "_index": ".ds-my-data-stream-2023.07.04-000001", "_id": "BHpTIYkBrVQ2QEgsWg94", "_source": { "@timestamp": "2099-12-07T11:07:10.000Z", "event": { "category": "file", "id": "tZ1NWVOs", "sequence": 5 }, "process": { "pid": 2012, "name": "regsvr32.exe", "executable": "C:\\Windows\\System32\\regsvr32.exe" }, "file": { "path": "C:\\Windows\\System32\\scrobj.dll", "name": "scrobj.dll" } } } ] } ] } }
使用 by
关键字来匹配共享相同字段值的事件
resp = client.eql.search( index="my-data-stream", query="\n sequence with maxspan=1h\n [ process where process.name == \"regsvr32.exe\" ] by process.pid\n [ file where stringContains(file.name, \"scrobj.dll\") ] by process.pid\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", query: '\n sequence with maxspan=1h\n [ process where process.name == "regsvr32.exe" ] by process.pid\n [ file where stringContains(file.name, "scrobj.dll") ] by process.pid\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "query": """ sequence with maxspan=1h [ process where process.name == "regsvr32.exe" ] by process.pid [ file where stringContains(file.name, "scrobj.dll") ] by process.pid """ }
如果字段值应在所有事件中共享,请使用 sequence by
关键字。以下查询与前一个查询等效。
resp = client.eql.search( index="my-data-stream", query="\n sequence by process.pid with maxspan=1h\n [ process where process.name == \"regsvr32.exe\" ]\n [ file where stringContains(file.name, \"scrobj.dll\") ]\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", query: '\n sequence by process.pid with maxspan=1h\n [ process where process.name == "regsvr32.exe" ]\n [ file where stringContains(file.name, "scrobj.dll") ]\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "query": """ sequence by process.pid with maxspan=1h [ process where process.name == "regsvr32.exe" ] [ file where stringContains(file.name, "scrobj.dll") ] """ }
hits.sequences.join_keys
属性包含共享的字段值。
{ ... "hits": ..., "sequences": [ { "join_keys": [ 2012 ], "events": ... } ] } }
使用 until
关键字来指定序列的过期事件。匹配序列必须在此事件之前结束。
resp = client.eql.search( index="my-data-stream", query="\n sequence by process.pid with maxspan=1h\n [ process where process.name == \"regsvr32.exe\" ]\n [ file where stringContains(file.name, \"scrobj.dll\") ]\n until [ process where event.type == \"termination\" ]\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", query: '\n sequence by process.pid with maxspan=1h\n [ process where process.name == "regsvr32.exe" ]\n [ file where stringContains(file.name, "scrobj.dll") ]\n until [ process where event.type == "termination" ]\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "query": """ sequence by process.pid with maxspan=1h [ process where process.name == "regsvr32.exe" ] [ file where stringContains(file.name, "scrobj.dll") ] until [ process where event.type == "termination" ] """ }
按时间顺序排列的无序事件示例
编辑使用 EQL 的 示例语法 来搜索匹配一个或多个连接键和一组过滤器的事件。示例类似于序列,但不按时间顺序返回事件。实际上,示例查询可以在没有时间戳的数据上运行。示例查询对于查找并非始终按相同顺序发生或跨越长时间跨度发生的事件中的相关性非常有用。
单击以显示以下示例中使用的示例数据
resp = client.indices.create( index="my-index-000001", mappings={ "properties": { "ip": { "type": "ip" }, "version": { "type": "version" }, "missing_keyword": { "type": "keyword" }, "@timestamp": { "type": "date" }, "type_test": { "type": "keyword" }, "@timestamp_pretty": { "type": "date", "format": "dd-MM-yyyy" }, "event_type": { "type": "keyword" }, "event": { "properties": { "category": { "type": "alias", "path": "event_type" } } }, "host": { "type": "keyword" }, "os": { "type": "keyword" }, "bool": { "type": "boolean" }, "uptime": { "type": "long" }, "port": { "type": "long" } } }, ) print(resp) resp1 = client.indices.create( index="my-index-000002", mappings={ "properties": { "ip": { "type": "ip" }, "@timestamp": { "type": "date" }, "@timestamp_pretty": { "type": "date", "format": "yyyy-MM-dd" }, "type_test": { "type": "keyword" }, "event_type": { "type": "keyword" }, "event": { "properties": { "category": { "type": "alias", "path": "event_type" } } }, "host": { "type": "keyword" }, "op_sys": { "type": "keyword" }, "bool": { "type": "boolean" }, "uptime": { "type": "long" }, "port": { "type": "long" } } }, ) print(resp1) resp2 = client.indices.create( index="my-index-000003", mappings={ "properties": { "host_ip": { "type": "ip" }, "@timestamp": { "type": "date" }, "date": { "type": "date" }, "event_type": { "type": "keyword" }, "event": { "properties": { "category": { "type": "alias", "path": "event_type" } } }, "missing_keyword": { "type": "keyword" }, "host": { "type": "keyword" }, "os": { "type": "keyword" }, "bool": { "type": "boolean" }, "uptime": { "type": "long" }, "port": { "type": "long" } } }, ) print(resp2) resp3 = client.bulk( index="my-index-000001", refresh=True, operations=[ { "index": { "_id": 1 } }, { "@timestamp": "1234567891", "@timestamp_pretty": "12-12-2022", "missing_keyword": "test", "type_test": "abc", "ip": "10.0.0.1", "event_type": "alert", "host": "doom", "uptime": 0, "port": 1234, "os": "win10", "version": "1.0.0", "id": 11 }, { "index": { "_id": 2 } }, { "@timestamp": "1234567892", "@timestamp_pretty": "13-12-2022", "event_type": "alert", "type_test": "abc", "host": "CS", "uptime": 5, "port": 1, "os": "win10", "version": "1.2.0", "id": 12 }, { "index": { "_id": 3 } }, { "@timestamp": "1234567893", "@timestamp_pretty": "12-12-2022", "event_type": "alert", "type_test": "abc", "host": "farcry", "uptime": 1, "port": 1234, "bool": False, "os": "win10", "version": "2.0.0", "id": 13 }, { "index": { "_id": 4 } }, { "@timestamp": "1234567894", "@timestamp_pretty": "13-12-2022", "event_type": "alert", "type_test": "abc", "host": "GTA", "uptime": 3, "port": 12, "os": "slack", "version": "10.0.0", "id": 14 }, { "index": { "_id": 5 } }, { "@timestamp": "1234567895", "@timestamp_pretty": "17-12-2022", "event_type": "alert", "host": "sniper 3d", "uptime": 6, "port": 1234, "os": "fedora", "version": "20.1.0", "id": 15 }, { "index": { "_id": 6 } }, { "@timestamp": "1234568896", "@timestamp_pretty": "17-12-2022", "event_type": "alert", "host": "doom", "port": 65123, "bool": True, "os": "redhat", "version": "20.10.0", "id": 16 }, { "index": { "_id": 7 } }, { "@timestamp": "1234567897", "@timestamp_pretty": "17-12-2022", "missing_keyword": "yyy", "event_type": "failure", "host": "doom", "uptime": 15, "port": 1234, "bool": True, "os": "redhat", "version": "20.2.0", "id": 17 }, { "index": { "_id": 8 } }, { "@timestamp": "1234567898", "@timestamp_pretty": "12-12-2022", "missing_keyword": "test", "event_type": "success", "host": "doom", "uptime": 16, "port": 512, "os": "win10", "version": "1.2.3", "id": 18 }, { "index": { "_id": 9 } }, { "@timestamp": "1234567899", "@timestamp_pretty": "15-12-2022", "missing_keyword": "test", "event_type": "success", "host": "GTA", "port": 12, "bool": True, "os": "win10", "version": "1.2.3", "id": 19 }, { "index": { "_id": 10 } }, { "@timestamp": "1234567893", "missing_keyword": None, "ip": "10.0.0.5", "event_type": "alert", "host": "farcry", "uptime": 1, "port": 1234, "bool": True, "os": "win10", "version": "1.2.3", "id": 110 } ], ) print(resp3) resp4 = client.bulk( index="my-index-000002", refresh=True, operations=[ { "index": { "_id": 1 } }, { "@timestamp": "1234567991", "type_test": "abc", "ip": "10.0.0.1", "event_type": "alert", "host": "doom", "uptime": 0, "port": 1234, "op_sys": "win10", "id": 21 }, { "index": { "_id": 2 } }, { "@timestamp": "1234567992", "type_test": "abc", "event_type": "alert", "host": "CS", "uptime": 5, "port": 1, "op_sys": "win10", "id": 22 }, { "index": { "_id": 3 } }, { "@timestamp": "1234567993", "type_test": "abc", "@timestamp_pretty": "2022-12-17", "event_type": "alert", "host": "farcry", "uptime": 1, "port": 1234, "bool": False, "op_sys": "win10", "id": 23 }, { "index": { "_id": 4 } }, { "@timestamp": "1234567994", "event_type": "alert", "host": "GTA", "uptime": 3, "port": 12, "op_sys": "slack", "id": 24 }, { "index": { "_id": 5 } }, { "@timestamp": "1234567995", "event_type": "alert", "host": "sniper 3d", "uptime": 6, "port": 1234, "op_sys": "fedora", "id": 25 }, { "index": { "_id": 6 } }, { "@timestamp": "1234568996", "@timestamp_pretty": "2022-12-17", "ip": "10.0.0.5", "event_type": "alert", "host": "doom", "port": 65123, "bool": True, "op_sys": "redhat", "id": 26 }, { "index": { "_id": 7 } }, { "@timestamp": "1234567997", "@timestamp_pretty": "2022-12-17", "event_type": "failure", "host": "doom", "uptime": 15, "port": 1234, "bool": True, "op_sys": "redhat", "id": 27 }, { "index": { "_id": 8 } }, { "@timestamp": "1234567998", "ip": "10.0.0.1", "event_type": "success", "host": "doom", "uptime": 16, "port": 512, "op_sys": "win10", "id": 28 }, { "index": { "_id": 9 } }, { "@timestamp": "1234567999", "ip": "10.0.0.1", "event_type": "success", "host": "GTA", "port": 12, "bool": False, "op_sys": "win10", "id": 29 } ], ) print(resp4) resp5 = client.bulk( index="my-index-000003", refresh=True, operations=[ { "index": { "_id": 1 } }, { "@timestamp": "1334567891", "host_ip": "10.0.0.1", "event_type": "alert", "host": "doom", "uptime": 0, "port": 12, "os": "win10", "id": 31 }, { "index": { "_id": 2 } }, { "@timestamp": "1334567892", "event_type": "alert", "host": "CS", "os": "win10", "id": 32 }, { "index": { "_id": 3 } }, { "@timestamp": "1334567893", "event_type": "alert", "host": "farcry", "bool": True, "os": "win10", "id": 33 }, { "index": { "_id": 4 } }, { "@timestamp": "1334567894", "event_type": "alert", "host": "GTA", "os": "slack", "bool": True, "id": 34 }, { "index": { "_id": 5 } }, { "@timestamp": "1234567895", "event_type": "alert", "host": "sniper 3d", "os": "fedora", "id": 35 }, { "index": { "_id": 6 } }, { "@timestamp": "1234578896", "host_ip": "10.0.0.1", "event_type": "alert", "host": "doom", "bool": True, "os": "redhat", "id": 36 }, { "index": { "_id": 7 } }, { "@timestamp": "1234567897", "event_type": "failure", "missing_keyword": "test", "host": "doom", "bool": True, "os": "redhat", "id": 37 }, { "index": { "_id": 8 } }, { "@timestamp": "1234577898", "event_type": "success", "host": "doom", "os": "win10", "id": 38, "date": "1671235200000" }, { "index": { "_id": 9 } }, { "@timestamp": "1234577899", "host_ip": "10.0.0.5", "event_type": "success", "host": "GTA", "bool": True, "os": "win10", "id": 39 } ], ) print(resp5)
response = client.indices.create( index: 'my-index-000001', body: { mappings: { properties: { ip: { type: 'ip' }, version: { type: 'version' }, missing_keyword: { type: 'keyword' }, "@timestamp": { type: 'date' }, type_test: { type: 'keyword' }, "@timestamp_pretty": { type: 'date', format: 'dd-MM-yyyy' }, event_type: { type: 'keyword' }, event: { properties: { category: { type: 'alias', path: 'event_type' } } }, host: { type: 'keyword' }, os: { type: 'keyword' }, bool: { type: 'boolean' }, uptime: { type: 'long' }, port: { type: 'long' } } } } ) puts response response = client.indices.create( index: 'my-index-000002', body: { mappings: { properties: { ip: { type: 'ip' }, "@timestamp": { type: 'date' }, "@timestamp_pretty": { type: 'date', format: 'yyyy-MM-dd' }, type_test: { type: 'keyword' }, event_type: { type: 'keyword' }, event: { properties: { category: { type: 'alias', path: 'event_type' } } }, host: { type: 'keyword' }, op_sys: { type: 'keyword' }, bool: { type: 'boolean' }, uptime: { type: 'long' }, port: { type: 'long' } } } } ) puts response response = client.indices.create( index: 'my-index-000003', body: { mappings: { properties: { host_ip: { type: 'ip' }, "@timestamp": { type: 'date' }, date: { type: 'date' }, event_type: { type: 'keyword' }, event: { properties: { category: { type: 'alias', path: 'event_type' } } }, missing_keyword: { type: 'keyword' }, host: { type: 'keyword' }, os: { type: 'keyword' }, bool: { type: 'boolean' }, uptime: { type: 'long' }, port: { type: 'long' } } } } ) puts response response = client.bulk( index: 'my-index-000001', refresh: true, body: [ { index: { _id: 1 } }, { "@timestamp": '1234567891', "@timestamp_pretty": '12-12-2022', missing_keyword: 'test', type_test: 'abc', ip: '10.0.0.1', event_type: 'alert', host: 'doom', uptime: 0, port: 1234, os: 'win10', version: '1.0.0', id: 11 }, { index: { _id: 2 } }, { "@timestamp": '1234567892', "@timestamp_pretty": '13-12-2022', event_type: 'alert', type_test: 'abc', host: 'CS', uptime: 5, port: 1, os: 'win10', version: '1.2.0', id: 12 }, { index: { _id: 3 } }, { "@timestamp": '1234567893', "@timestamp_pretty": '12-12-2022', event_type: 'alert', type_test: 'abc', host: 'farcry', uptime: 1, port: 1234, bool: false, os: 'win10', version: '2.0.0', id: 13 }, { index: { _id: 4 } }, { "@timestamp": '1234567894', "@timestamp_pretty": '13-12-2022', event_type: 'alert', type_test: 'abc', host: 'GTA', uptime: 3, port: 12, os: 'slack', version: '10.0.0', id: 14 }, { index: { _id: 5 } }, { "@timestamp": '1234567895', "@timestamp_pretty": '17-12-2022', event_type: 'alert', host: 'sniper 3d', uptime: 6, port: 1234, os: 'fedora', version: '20.1.0', id: 15 }, { index: { _id: 6 } }, { "@timestamp": '1234568896', "@timestamp_pretty": '17-12-2022', event_type: 'alert', host: 'doom', port: 65_123, bool: true, os: 'redhat', version: '20.10.0', id: 16 }, { index: { _id: 7 } }, { "@timestamp": '1234567897', "@timestamp_pretty": '17-12-2022', missing_keyword: 'yyy', event_type: 'failure', host: 'doom', uptime: 15, port: 1234, bool: true, os: 'redhat', version: '20.2.0', id: 17 }, { index: { _id: 8 } }, { "@timestamp": '1234567898', "@timestamp_pretty": '12-12-2022', missing_keyword: 'test', event_type: 'success', host: 'doom', uptime: 16, port: 512, os: 'win10', version: '1.2.3', id: 18 }, { index: { _id: 9 } }, { "@timestamp": '1234567899', "@timestamp_pretty": '15-12-2022', missing_keyword: 'test', event_type: 'success', host: 'GTA', port: 12, bool: true, os: 'win10', version: '1.2.3', id: 19 }, { index: { _id: 10 } }, { "@timestamp": '1234567893', missing_keyword: nil, ip: '10.0.0.5', event_type: 'alert', host: 'farcry', uptime: 1, port: 1234, bool: true, os: 'win10', version: '1.2.3', id: 110 } ] ) puts response response = client.bulk( index: 'my-index-000002', refresh: true, body: [ { index: { _id: 1 } }, { "@timestamp": '1234567991', type_test: 'abc', ip: '10.0.0.1', event_type: 'alert', host: 'doom', uptime: 0, port: 1234, op_sys: 'win10', id: 21 }, { index: { _id: 2 } }, { "@timestamp": '1234567992', type_test: 'abc', event_type: 'alert', host: 'CS', uptime: 5, port: 1, op_sys: 'win10', id: 22 }, { index: { _id: 3 } }, { "@timestamp": '1234567993', type_test: 'abc', "@timestamp_pretty": '2022-12-17', event_type: 'alert', host: 'farcry', uptime: 1, port: 1234, bool: false, op_sys: 'win10', id: 23 }, { index: { _id: 4 } }, { "@timestamp": '1234567994', event_type: 'alert', host: 'GTA', uptime: 3, port: 12, op_sys: 'slack', id: 24 }, { index: { _id: 5 } }, { "@timestamp": '1234567995', event_type: 'alert', host: 'sniper 3d', uptime: 6, port: 1234, op_sys: 'fedora', id: 25 }, { index: { _id: 6 } }, { "@timestamp": '1234568996', "@timestamp_pretty": '2022-12-17', ip: '10.0.0.5', event_type: 'alert', host: 'doom', port: 65_123, bool: true, op_sys: 'redhat', id: 26 }, { index: { _id: 7 } }, { "@timestamp": '1234567997', "@timestamp_pretty": '2022-12-17', event_type: 'failure', host: 'doom', uptime: 15, port: 1234, bool: true, op_sys: 'redhat', id: 27 }, { index: { _id: 8 } }, { "@timestamp": '1234567998', ip: '10.0.0.1', event_type: 'success', host: 'doom', uptime: 16, port: 512, op_sys: 'win10', id: 28 }, { index: { _id: 9 } }, { "@timestamp": '1234567999', ip: '10.0.0.1', event_type: 'success', host: 'GTA', port: 12, bool: false, op_sys: 'win10', id: 29 } ] ) puts response response = client.bulk( index: 'my-index-000003', refresh: true, body: [ { index: { _id: 1 } }, { "@timestamp": '1334567891', host_ip: '10.0.0.1', event_type: 'alert', host: 'doom', uptime: 0, port: 12, os: 'win10', id: 31 }, { index: { _id: 2 } }, { "@timestamp": '1334567892', event_type: 'alert', host: 'CS', os: 'win10', id: 32 }, { index: { _id: 3 } }, { "@timestamp": '1334567893', event_type: 'alert', host: 'farcry', bool: true, os: 'win10', id: 33 }, { index: { _id: 4 } }, { "@timestamp": '1334567894', event_type: 'alert', host: 'GTA', os: 'slack', bool: true, id: 34 }, { index: { _id: 5 } }, { "@timestamp": '1234567895', event_type: 'alert', host: 'sniper 3d', os: 'fedora', id: 35 }, { index: { _id: 6 } }, { "@timestamp": '1234578896', host_ip: '10.0.0.1', event_type: 'alert', host: 'doom', bool: true, os: 'redhat', id: 36 }, { index: { _id: 7 } }, { "@timestamp": '1234567897', event_type: 'failure', missing_keyword: 'test', host: 'doom', bool: true, os: 'redhat', id: 37 }, { index: { _id: 8 } }, { "@timestamp": '1234577898', event_type: 'success', host: 'doom', os: 'win10', id: 38, date: '1671235200000' }, { index: { _id: 9 } }, { "@timestamp": '1234577899', host_ip: '10.0.0.5', event_type: 'success', host: 'GTA', bool: true, os: 'win10', id: 39 } ] ) puts response
const response = await client.indices.create({ index: "my-index-000001", mappings: { properties: { ip: { type: "ip", }, version: { type: "version", }, missing_keyword: { type: "keyword", }, "@timestamp": { type: "date", }, type_test: { type: "keyword", }, "@timestamp_pretty": { type: "date", format: "dd-MM-yyyy", }, event_type: { type: "keyword", }, event: { properties: { category: { type: "alias", path: "event_type", }, }, }, host: { type: "keyword", }, os: { type: "keyword", }, bool: { type: "boolean", }, uptime: { type: "long", }, port: { type: "long", }, }, }, }); console.log(response); const response1 = await client.indices.create({ index: "my-index-000002", mappings: { properties: { ip: { type: "ip", }, "@timestamp": { type: "date", }, "@timestamp_pretty": { type: "date", format: "yyyy-MM-dd", }, type_test: { type: "keyword", }, event_type: { type: "keyword", }, event: { properties: { category: { type: "alias", path: "event_type", }, }, }, host: { type: "keyword", }, op_sys: { type: "keyword", }, bool: { type: "boolean", }, uptime: { type: "long", }, port: { type: "long", }, }, }, }); console.log(response1); const response2 = await client.indices.create({ index: "my-index-000003", mappings: { properties: { host_ip: { type: "ip", }, "@timestamp": { type: "date", }, date: { type: "date", }, event_type: { type: "keyword", }, event: { properties: { category: { type: "alias", path: "event_type", }, }, }, missing_keyword: { type: "keyword", }, host: { type: "keyword", }, os: { type: "keyword", }, bool: { type: "boolean", }, uptime: { type: "long", }, port: { type: "long", }, }, }, }); console.log(response2); const response3 = await client.bulk({ index: "my-index-000001", refresh: "true", operations: [ { index: { _id: 1, }, }, { "@timestamp": "1234567891", "@timestamp_pretty": "12-12-2022", missing_keyword: "test", type_test: "abc", ip: "10.0.0.1", event_type: "alert", host: "doom", uptime: 0, port: 1234, os: "win10", version: "1.0.0", id: 11, }, { index: { _id: 2, }, }, { "@timestamp": "1234567892", "@timestamp_pretty": "13-12-2022", event_type: "alert", type_test: "abc", host: "CS", uptime: 5, port: 1, os: "win10", version: "1.2.0", id: 12, }, { index: { _id: 3, }, }, { "@timestamp": "1234567893", "@timestamp_pretty": "12-12-2022", event_type: "alert", type_test: "abc", host: "farcry", uptime: 1, port: 1234, bool: false, os: "win10", version: "2.0.0", id: 13, }, { index: { _id: 4, }, }, { "@timestamp": "1234567894", "@timestamp_pretty": "13-12-2022", event_type: "alert", type_test: "abc", host: "GTA", uptime: 3, port: 12, os: "slack", version: "10.0.0", id: 14, }, { index: { _id: 5, }, }, { "@timestamp": "1234567895", "@timestamp_pretty": "17-12-2022", event_type: "alert", host: "sniper 3d", uptime: 6, port: 1234, os: "fedora", version: "20.1.0", id: 15, }, { index: { _id: 6, }, }, { "@timestamp": "1234568896", "@timestamp_pretty": "17-12-2022", event_type: "alert", host: "doom", port: 65123, bool: true, os: "redhat", version: "20.10.0", id: 16, }, { index: { _id: 7, }, }, { "@timestamp": "1234567897", "@timestamp_pretty": "17-12-2022", missing_keyword: "yyy", event_type: "failure", host: "doom", uptime: 15, port: 1234, bool: true, os: "redhat", version: "20.2.0", id: 17, }, { index: { _id: 8, }, }, { "@timestamp": "1234567898", "@timestamp_pretty": "12-12-2022", missing_keyword: "test", event_type: "success", host: "doom", uptime: 16, port: 512, os: "win10", version: "1.2.3", id: 18, }, { index: { _id: 9, }, }, { "@timestamp": "1234567899", "@timestamp_pretty": "15-12-2022", missing_keyword: "test", event_type: "success", host: "GTA", port: 12, bool: true, os: "win10", version: "1.2.3", id: 19, }, { index: { _id: 10, }, }, { "@timestamp": "1234567893", missing_keyword: null, ip: "10.0.0.5", event_type: "alert", host: "farcry", uptime: 1, port: 1234, bool: true, os: "win10", version: "1.2.3", id: 110, }, ], }); console.log(response3); const response4 = await client.bulk({ index: "my-index-000002", refresh: "true", operations: [ { index: { _id: 1, }, }, { "@timestamp": "1234567991", type_test: "abc", ip: "10.0.0.1", event_type: "alert", host: "doom", uptime: 0, port: 1234, op_sys: "win10", id: 21, }, { index: { _id: 2, }, }, { "@timestamp": "1234567992", type_test: "abc", event_type: "alert", host: "CS", uptime: 5, port: 1, op_sys: "win10", id: 22, }, { index: { _id: 3, }, }, { "@timestamp": "1234567993", type_test: "abc", "@timestamp_pretty": "2022-12-17", event_type: "alert", host: "farcry", uptime: 1, port: 1234, bool: false, op_sys: "win10", id: 23, }, { index: { _id: 4, }, }, { "@timestamp": "1234567994", event_type: "alert", host: "GTA", uptime: 3, port: 12, op_sys: "slack", id: 24, }, { index: { _id: 5, }, }, { "@timestamp": "1234567995", event_type: "alert", host: "sniper 3d", uptime: 6, port: 1234, op_sys: "fedora", id: 25, }, { index: { _id: 6, }, }, { "@timestamp": "1234568996", "@timestamp_pretty": "2022-12-17", ip: "10.0.0.5", event_type: "alert", host: "doom", port: 65123, bool: true, op_sys: "redhat", id: 26, }, { index: { _id: 7, }, }, { "@timestamp": "1234567997", "@timestamp_pretty": "2022-12-17", event_type: "failure", host: "doom", uptime: 15, port: 1234, bool: true, op_sys: "redhat", id: 27, }, { index: { _id: 8, }, }, { "@timestamp": "1234567998", ip: "10.0.0.1", event_type: "success", host: "doom", uptime: 16, port: 512, op_sys: "win10", id: 28, }, { index: { _id: 9, }, }, { "@timestamp": "1234567999", ip: "10.0.0.1", event_type: "success", host: "GTA", port: 12, bool: false, op_sys: "win10", id: 29, }, ], }); console.log(response4); const response5 = await client.bulk({ index: "my-index-000003", refresh: "true", operations: [ { index: { _id: 1, }, }, { "@timestamp": "1334567891", host_ip: "10.0.0.1", event_type: "alert", host: "doom", uptime: 0, port: 12, os: "win10", id: 31, }, { index: { _id: 2, }, }, { "@timestamp": "1334567892", event_type: "alert", host: "CS", os: "win10", id: 32, }, { index: { _id: 3, }, }, { "@timestamp": "1334567893", event_type: "alert", host: "farcry", bool: true, os: "win10", id: 33, }, { index: { _id: 4, }, }, { "@timestamp": "1334567894", event_type: "alert", host: "GTA", os: "slack", bool: true, id: 34, }, { index: { _id: 5, }, }, { "@timestamp": "1234567895", event_type: "alert", host: "sniper 3d", os: "fedora", id: 35, }, { index: { _id: 6, }, }, { "@timestamp": "1234578896", host_ip: "10.0.0.1", event_type: "alert", host: "doom", bool: true, os: "redhat", id: 36, }, { index: { _id: 7, }, }, { "@timestamp": "1234567897", event_type: "failure", missing_keyword: "test", host: "doom", bool: true, os: "redhat", id: 37, }, { index: { _id: 8, }, }, { "@timestamp": "1234577898", event_type: "success", host: "doom", os: "win10", id: 38, date: "1671235200000", }, { index: { _id: 9, }, }, { "@timestamp": "1234577899", host_ip: "10.0.0.5", event_type: "success", host: "GTA", bool: true, os: "win10", id: 39, }, ], }); console.log(response5);
PUT /my-index-000001 { "mappings": { "properties": { "ip": { "type":"ip" }, "version": { "type": "version" }, "missing_keyword": { "type": "keyword" }, "@timestamp": { "type": "date" }, "type_test": { "type": "keyword" }, "@timestamp_pretty": { "type": "date", "format": "dd-MM-yyyy" }, "event_type": { "type": "keyword" }, "event": { "properties": { "category": { "type": "alias", "path": "event_type" } } }, "host": { "type": "keyword" }, "os": { "type": "keyword" }, "bool": { "type": "boolean" }, "uptime" : { "type" : "long" }, "port" : { "type" : "long" } } } } PUT /my-index-000002 { "mappings": { "properties": { "ip": { "type":"ip" }, "@timestamp": { "type": "date" }, "@timestamp_pretty": { "type": "date", "format": "yyyy-MM-dd" }, "type_test": { "type": "keyword" }, "event_type": { "type": "keyword" }, "event": { "properties": { "category": { "type": "alias", "path": "event_type" } } }, "host": { "type": "keyword" }, "op_sys": { "type": "keyword" }, "bool": { "type": "boolean" }, "uptime" : { "type" : "long" }, "port" : { "type" : "long" } } } } PUT /my-index-000003 { "mappings": { "properties": { "host_ip": { "type":"ip" }, "@timestamp": { "type": "date" }, "date": { "type": "date" }, "event_type": { "type": "keyword" }, "event": { "properties": { "category": { "type": "alias", "path": "event_type" } } }, "missing_keyword": { "type": "keyword" }, "host": { "type": "keyword" }, "os": { "type": "keyword" }, "bool": { "type": "boolean" }, "uptime" : { "type" : "long" }, "port" : { "type" : "long" } } } } POST /my-index-000001/_bulk?refresh {"index":{"_id":1}} {"@timestamp":"1234567891","@timestamp_pretty":"12-12-2022","missing_keyword":"test","type_test":"abc","ip":"10.0.0.1","event_type":"alert","host":"doom","uptime":0,"port":1234,"os":"win10","version":"1.0.0","id":11} {"index":{"_id":2}} {"@timestamp":"1234567892","@timestamp_pretty":"13-12-2022","event_type":"alert","type_test":"abc","host":"CS","uptime":5,"port":1,"os":"win10","version":"1.2.0","id":12} {"index":{"_id":3}} {"@timestamp":"1234567893","@timestamp_pretty":"12-12-2022","event_type":"alert","type_test":"abc","host":"farcry","uptime":1,"port":1234,"bool":false,"os":"win10","version":"2.0.0","id":13} {"index":{"_id":4}} {"@timestamp":"1234567894","@timestamp_pretty":"13-12-2022","event_type":"alert","type_test":"abc","host":"GTA","uptime":3,"port":12,"os":"slack","version":"10.0.0","id":14} {"index":{"_id":5}} {"@timestamp":"1234567895","@timestamp_pretty":"17-12-2022","event_type":"alert","host":"sniper 3d","uptime":6,"port":1234,"os":"fedora","version":"20.1.0","id":15} {"index":{"_id":6}} {"@timestamp":"1234568896","@timestamp_pretty":"17-12-2022","event_type":"alert","host":"doom","port":65123,"bool":true,"os":"redhat","version":"20.10.0","id":16} {"index":{"_id":7}} {"@timestamp":"1234567897","@timestamp_pretty":"17-12-2022","missing_keyword":"yyy","event_type":"failure","host":"doom","uptime":15,"port":1234,"bool":true,"os":"redhat","version":"20.2.0","id":17} {"index":{"_id":8}} {"@timestamp":"1234567898","@timestamp_pretty":"12-12-2022","missing_keyword":"test","event_type":"success","host":"doom","uptime":16,"port":512,"os":"win10","version":"1.2.3","id":18} {"index":{"_id":9}} {"@timestamp":"1234567899","@timestamp_pretty":"15-12-2022","missing_keyword":"test","event_type":"success","host":"GTA","port":12,"bool":true,"os":"win10","version":"1.2.3","id":19} {"index":{"_id":10}} {"@timestamp":"1234567893","missing_keyword":null,"ip":"10.0.0.5","event_type":"alert","host":"farcry","uptime":1,"port":1234,"bool":true,"os":"win10","version":"1.2.3","id":110} POST /my-index-000002/_bulk?refresh {"index":{"_id":1}} {"@timestamp":"1234567991","type_test":"abc","ip":"10.0.0.1","event_type":"alert","host":"doom","uptime":0,"port":1234,"op_sys":"win10","id":21} {"index":{"_id":2}} {"@timestamp":"1234567992","type_test":"abc","event_type":"alert","host":"CS","uptime":5,"port":1,"op_sys":"win10","id":22} {"index":{"_id":3}} {"@timestamp":"1234567993","type_test":"abc","@timestamp_pretty":"2022-12-17","event_type":"alert","host":"farcry","uptime":1,"port":1234,"bool":false,"op_sys":"win10","id":23} {"index":{"_id":4}} {"@timestamp":"1234567994","event_type":"alert","host":"GTA","uptime":3,"port":12,"op_sys":"slack","id":24} {"index":{"_id":5}} {"@timestamp":"1234567995","event_type":"alert","host":"sniper 3d","uptime":6,"port":1234,"op_sys":"fedora","id":25} {"index":{"_id":6}} {"@timestamp":"1234568996","@timestamp_pretty":"2022-12-17","ip":"10.0.0.5","event_type":"alert","host":"doom","port":65123,"bool":true,"op_sys":"redhat","id":26} {"index":{"_id":7}} {"@timestamp":"1234567997","@timestamp_pretty":"2022-12-17","event_type":"failure","host":"doom","uptime":15,"port":1234,"bool":true,"op_sys":"redhat","id":27} {"index":{"_id":8}} {"@timestamp":"1234567998","ip":"10.0.0.1","event_type":"success","host":"doom","uptime":16,"port":512,"op_sys":"win10","id":28} {"index":{"_id":9}} {"@timestamp":"1234567999","ip":"10.0.0.1","event_type":"success","host":"GTA","port":12,"bool":false,"op_sys":"win10","id":29} POST /my-index-000003/_bulk?refresh {"index":{"_id":1}} {"@timestamp":"1334567891","host_ip":"10.0.0.1","event_type":"alert","host":"doom","uptime":0,"port":12,"os":"win10","id":31} {"index":{"_id":2}} {"@timestamp":"1334567892","event_type":"alert","host":"CS","os":"win10","id":32} {"index":{"_id":3}} {"@timestamp":"1334567893","event_type":"alert","host":"farcry","bool":true,"os":"win10","id":33} {"index":{"_id":4}} {"@timestamp":"1334567894","event_type":"alert","host":"GTA","os":"slack","bool":true,"id":34} {"index":{"_id":5}} {"@timestamp":"1234567895","event_type":"alert","host":"sniper 3d","os":"fedora","id":35} {"index":{"_id":6}} {"@timestamp":"1234578896","host_ip":"10.0.0.1","event_type":"alert","host":"doom","bool":true,"os":"redhat","id":36} {"index":{"_id":7}} {"@timestamp":"1234567897","event_type":"failure","missing_keyword":"test","host":"doom","bool":true,"os":"redhat","id":37} {"index":{"_id":8}} {"@timestamp":"1234577898","event_type":"success","host":"doom","os":"win10","id":38,"date":"1671235200000"} {"index":{"_id":9}} {"@timestamp":"1234577899","host_ip":"10.0.0.5","event_type":"success","host":"GTA","bool":true,"os":"win10","id":39}
示例查询使用 by
关键字指定至少一个连接键,以及最多五个过滤器
resp = client.eql.search( index="my-index*", query="\n sample by host\n [any where uptime > 0]\n [any where port > 100]\n [any where bool == true]\n ", ) print(resp)
const response = await client.eql.search({ index: "my-index*", query: "\n sample by host\n [any where uptime > 0]\n [any where port > 100]\n [any where bool == true]\n ", }); console.log(response);
GET /my-index*/_eql/search { "query": """ sample by host [any where uptime > 0] [any where port > 100] [any where bool == true] """ }
默认情况下,响应的 hits.sequences
属性包含最多 10 个示例。每个示例都有一组 join_keys
和一个数组,该数组包含每个过滤器的匹配事件。事件按照它们匹配的过滤器顺序返回
{ ... "hits": { "total": { "value": 2, "relation": "eq" }, "sequences": [ { "join_keys": [ "doom" ], "events": [ { "_index": "my-index-000001", "_id": "7", "_source": { "@timestamp": "1234567897", "@timestamp_pretty": "17-12-2022", "missing_keyword": "yyy", "event_type": "failure", "host": "doom", "uptime": 15, "port": 1234, "bool": true, "os": "redhat", "version": "20.2.0", "id": 17 } }, { "_index": "my-index-000001", "_id": "1", "_source": { "@timestamp": "1234567891", "@timestamp_pretty": "12-12-2022", "missing_keyword": "test", "type_test": "abc", "ip": "10.0.0.1", "event_type": "alert", "host": "doom", "uptime": 0, "port": 1234, "os": "win10", "version": "1.0.0", "id": 11 } }, { "_index": "my-index-000001", "_id": "6", "_source": { "@timestamp": "1234568896", "@timestamp_pretty": "17-12-2022", "event_type": "alert", "host": "doom", "port": 65123, "bool": true, "os": "redhat", "version": "20.10.0", "id": 16 } } ] }, { "join_keys": [ "farcry" ], "events": [ { "_index": "my-index-000001", "_id": "3", "_source": { "@timestamp": "1234567893", "@timestamp_pretty": "12-12-2022", "event_type": "alert", "type_test": "abc", "host": "farcry", "uptime": 1, "port": 1234, "bool": false, "os": "win10", "version": "2.0.0", "id": 13 } }, { "_index": "my-index-000001", "_id": "10", "_source": { "@timestamp": "1234567893", "missing_keyword": null, "ip": "10.0.0.5", "event_type": "alert", "host": "farcry", "uptime": 1, "port": 1234, "bool": true, "os": "win10", "version": "1.2.3", "id": 110 } }, { "_index": "my-index-000003", "_id": "3", "_source": { "@timestamp": "1334567893", "event_type": "alert", "host": "farcry", "bool": true, "os": "win10", "id": 33 } } ] } ] } }
您可以指定多个连接键
resp = client.eql.search( index="my-index*", query="\n sample by host\n [any where uptime > 0] by os\n [any where port > 100] by op_sys\n [any where bool == true] by os\n ", ) print(resp)
const response = await client.eql.search({ index: "my-index*", query: "\n sample by host\n [any where uptime > 0] by os\n [any where port > 100] by op_sys\n [any where bool == true] by os\n ", }); console.log(response);
GET /my-index*/_eql/search { "query": """ sample by host [any where uptime > 0] by os [any where port > 100] by op_sys [any where bool == true] by os """ }
此查询将返回示例,其中每个事件对于 os
或 op_sys
以及 host
共享相同的值。例如
{ ... "hits": { "total": { "value": 2, "relation": "eq" }, "sequences": [ { "join_keys": [ "doom", "redhat" ], "events": [ { "_index": "my-index-000001", "_id": "7", "_source": { "@timestamp": "1234567897", "@timestamp_pretty": "17-12-2022", "missing_keyword": "yyy", "event_type": "failure", "host": "doom", "uptime": 15, "port": 1234, "bool": true, "os": "redhat", "version": "20.2.0", "id": 17 } }, { "_index": "my-index-000002", "_id": "6", "_source": { "@timestamp": "1234568996", "@timestamp_pretty": "2022-12-17", "ip": "10.0.0.5", "event_type": "alert", "host": "doom", "port": 65123, "bool": true, "op_sys": "redhat", "id": 26 } }, { "_index": "my-index-000001", "_id": "6", "_source": { "@timestamp": "1234568896", "@timestamp_pretty": "17-12-2022", "event_type": "alert", "host": "doom", "port": 65123, "bool": true, "os": "redhat", "version": "20.10.0", "id": 16 } } ] }, { "join_keys": [ "farcry", "win10" ], "events": [ { "_index": "my-index-000001", "_id": "3", "_source": { "@timestamp": "1234567893", "@timestamp_pretty": "12-12-2022", "event_type": "alert", "type_test": "abc", "host": "farcry", "uptime": 1, "port": 1234, "bool": false, "os": "win10", "version": "2.0.0", "id": 13 } }, { "_index": "my-index-000002", "_id": "3", "_source": { "@timestamp": "1234567993", "type_test": "abc", "@timestamp_pretty": "2022-12-17", "event_type": "alert", "host": "farcry", "uptime": 1, "port": 1234, "bool": false, "op_sys": "win10", "id": 23 } }, { "_index": "my-index-000001", "_id": "10", "_source": { "@timestamp": "1234567893", "missing_keyword": null, "ip": "10.0.0.5", "event_type": "alert", "host": "farcry", "uptime": 1, "port": 1234, "bool": true, "os": "win10", "version": "1.2.3", "id": 110 } } ] } ] } }
默认情况下,示例查询的响应包含最多 10 个示例,每个唯一连接键集对应一个示例。使用 size
参数来获取较小或较大的示例集。要检索每个连接键集的多个示例,请使用 max_samples_per_key
参数。管道不支持用于示例查询。
resp = client.eql.search( index="my-index*", max_samples_per_key=2, size=20, query="\n sample\n [any where uptime > 0] by host,os\n [any where port > 100] by host,op_sys\n [any where bool == true] by host,os\n ", ) print(resp)
const response = await client.eql.search({ index: "my-index*", max_samples_per_key: 2, size: 20, query: "\n sample\n [any where uptime > 0] by host,os\n [any where port > 100] by host,op_sys\n [any where bool == true] by host,os\n ", }); console.log(response);
GET /my-index*/_eql/search { "max_samples_per_key": 2, "size": 20, "query": """ sample [any where uptime > 0] by host,os [any where port > 100] by host,op_sys [any where bool == true] by host,os """ }
检索选定的字段
编辑默认情况下,搜索响应中的每次命中都包含文档 _source
,它是索引文档时提供的整个 JSON 对象。
您可以使用 filter_path
查询参数来筛选 API 响应。例如,以下搜索仅从每个匹配事件的 _source
返回时间戳和 PID。
resp = client.eql.search( index="my-data-stream", filter_path="hits.events._source.@timestamp,hits.events._source.process.pid", query="\n process where process.name == \"regsvr32.exe\"\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", filter_path: "hits.events._source.@timestamp,hits.events._source.process.pid", query: '\n process where process.name == "regsvr32.exe"\n ', }); console.log(response);
GET /my-data-stream/_eql/search?filter_path=hits.events._source.@timestamp,hits.events._source.process.pid { "query": """ process where process.name == "regsvr32.exe" """ }
API 返回以下响应。
{ "hits": { "events": [ { "_source": { "@timestamp": "2099-12-07T11:07:09.000Z", "process": { "pid": 2012 } } }, { "_source": { "@timestamp": "2099-12-07T11:07:10.000Z", "process": { "pid": 2012 } } } ] } }
您还可以使用 fields
参数来检索和格式化响应中的特定字段。此字段与搜索 API 的 fields
参数相同。
由于它会查询索引映射,因此 fields
参数比直接引用 _source
提供了几个优势。具体来说,fields
参数
以下搜索请求使用 fields
参数来检索 event.type
字段、所有以 process.
开头的字段和 @timestamp
字段的值。该请求还使用 filter_path
查询参数来排除每次命中的 _source
。
resp = client.eql.search( index="my-data-stream", filter_path="-hits.events._source", query="\n process where process.name == \"regsvr32.exe\"\n ", fields=[ "event.type", "process.*", { "field": "@timestamp", "format": "epoch_millis" } ], ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", filter_path: "-hits.events._source", query: '\n process where process.name == "regsvr32.exe"\n ', fields: [ "event.type", "process.*", { field: "@timestamp", format: "epoch_millis", }, ], }); console.log(response);
GET /my-data-stream/_eql/search?filter_path=-hits.events._source { "query": """ process where process.name == "regsvr32.exe" """, "fields": [ "event.type", "process.*", { "field": "@timestamp", "format": "epoch_millis" } ] }
响应在每次命中的 fields
部分中以平面列表形式包含值。
{ ... "hits": { "total": ..., "events": [ { "_index": ".ds-my-data-stream-2099.12.07-000001", "_id": "OQmfCaduce8zoHT93o4H", "fields": { "process.name": [ "regsvr32.exe" ], "process.name.keyword": [ "regsvr32.exe" ], "@timestamp": [ "4100324829000" ], "process.command_line": [ "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll" ], "process.command_line.keyword": [ "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll" ], "process.executable.keyword": [ "C:\\Windows\\System32\\regsvr32.exe" ], "process.pid": [ 2012 ], "process.executable": [ "C:\\Windows\\System32\\regsvr32.exe" ] } }, .... ] } }
使用运行时字段
编辑使用 runtime_mappings
参数在搜索期间提取和创建 运行时字段。使用 fields
参数将运行时字段包括在响应中。
以下搜索从 @timestamp
创建 day_of_week
运行时字段,并在响应中返回该字段。
resp = client.eql.search( index="my-data-stream", filter_path="-hits.events._source", runtime_mappings={ "day_of_week": { "type": "keyword", "script": "emit(doc['@timestamp'].value.dayOfWeekEnum.toString())" } }, query="\n process where process.name == \"regsvr32.exe\"\n ", fields=[ "@timestamp", "day_of_week" ], ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", filter_path: "-hits.events._source", runtime_mappings: { day_of_week: { type: "keyword", script: "emit(doc['@timestamp'].value.dayOfWeekEnum.toString())", }, }, query: '\n process where process.name == "regsvr32.exe"\n ', fields: ["@timestamp", "day_of_week"], }); console.log(response);
GET /my-data-stream/_eql/search?filter_path=-hits.events._source { "runtime_mappings": { "day_of_week": { "type": "keyword", "script": "emit(doc['@timestamp'].value.dayOfWeekEnum.toString())" } }, "query": """ process where process.name == "regsvr32.exe" """, "fields": [ "@timestamp", "day_of_week" ] }
API 返回
{ ... "hits": { "total": ..., "events": [ { "_index": ".ds-my-data-stream-2099.12.07-000001", "_id": "OQmfCaduce8zoHT93o4H", "fields": { "@timestamp": [ "2099-12-07T11:07:09.000Z" ], "day_of_week": [ "MONDAY" ] } }, .... ] } }
指定时间戳或事件类别字段
编辑EQL 搜索 API 默认使用来自 ECS 的 @timestamp
和 event.category
字段。要指定不同的字段,请使用 timestamp_field
和 event_category_field
参数
resp = client.eql.search( index="my-data-stream", timestamp_field="file.accessed", event_category_field="file.type", query="\n file where (file.size > 1 and file.type == \"file\")\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", timestamp_field: "file.accessed", event_category_field: "file.type", query: '\n file where (file.size > 1 and file.type == "file")\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "timestamp_field": "file.accessed", "event_category_field": "file.type", "query": """ file where (file.size > 1 and file.type == "file") """ }
事件类别字段必须映射为 keyword
系列字段类型。时间戳字段应映射为 date
字段类型。不支持 date_nanos
时间戳字段。您不能使用 nested
字段或 nested
字段的子字段作为时间戳或事件类别字段。
指定排序决胜器
编辑默认情况下,EQL 搜索 API 会按时间戳返回匹配的命中结果。如果两个或多个事件具有相同的时间戳,Elasticsearch 会使用一个决胜字段值按升序对事件进行排序。Elasticsearch 会将没有决胜值的事件排在有值的事件之后。
如果您没有指定决胜字段,或者事件也具有相同的决胜值,Elasticsearch 会认为这些事件是并发的,并且可能不会以一致的排序顺序返回它们。
要指定决胜字段,请使用 tiebreaker_field
参数。如果您使用 ECS,我们建议使用 event.sequence
作为决胜字段。
resp = client.eql.search( index="my-data-stream", tiebreaker_field="event.sequence", query="\n process where process.name == \"cmd.exe\" and stringContains(process.executable, \"System32\")\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", tiebreaker_field: "event.sequence", query: '\n process where process.name == "cmd.exe" and stringContains(process.executable, "System32")\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "tiebreaker_field": "event.sequence", "query": """ process where process.name == "cmd.exe" and stringContains(process.executable, "System32") """ }
使用查询 DSL 进行过滤
编辑filter
参数使用 查询 DSL 来限制 EQL 查询运行的文档范围。
resp = client.eql.search( index="my-data-stream", filter={ "range": { "@timestamp": { "gte": "now-1d/d", "lt": "now/d" } } }, query="\n file where (file.type == \"file\" and file.name == \"cmd.exe\")\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", filter: { range: { "@timestamp": { gte: "now-1d/d", lt: "now/d", }, }, }, query: '\n file where (file.type == "file" and file.name == "cmd.exe")\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "filter": { "range": { "@timestamp": { "gte": "now-1d/d", "lt": "now/d" } } }, "query": """ file where (file.type == "file" and file.name == "cmd.exe") """ }
运行异步 EQL 搜索
编辑默认情况下,EQL 搜索请求是同步的,并且在返回响应之前会等待完整的结果。但是,对于跨大型数据集或 冻结数据的搜索,完整结果可能需要更长时间。
为了避免长时间等待,请运行异步 EQL 搜索。将 wait_for_completion_timeout
设置为您希望等待同步结果的持续时间。
resp = client.eql.search( index="my-data-stream", wait_for_completion_timeout="2s", query="\n process where process.name == \"cmd.exe\"\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", wait_for_completion_timeout: "2s", query: '\n process where process.name == "cmd.exe"\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "wait_for_completion_timeout": "2s", "query": """ process where process.name == "cmd.exe" """ }
如果请求在超时时间内没有完成,搜索将变为异步,并返回一个包含以下内容的响应:
- 搜索 ID
- 一个
is_partial
值为true
,表示搜索结果不完整 - 一个
is_running
值为true
,表示搜索正在进行中
异步搜索将在后台继续运行,而不会阻塞其他请求。
{ "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", "is_partial": true, "is_running": true, "took": 2000, "timed_out": false, "hits": ... }
要检查异步搜索的进度,请使用带有搜索 ID 的 获取异步 EQL 搜索 API。在 wait_for_completion_timeout
参数中指定您希望等待完整结果的时间。
resp = client.eql.get( id="FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", wait_for_completion_timeout="2s", ) print(resp)
response = client.eql.get( id: 'FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=', wait_for_completion_timeout: '2s' ) puts response
const response = await client.eql.get({ id: "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", wait_for_completion_timeout: "2s", }); console.log(response);
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s
如果响应的 is_running
值为 false
,则表示异步搜索已完成。如果 is_partial
值为 false
,则表示返回的搜索结果是完整的。
{ "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", "is_partial": false, "is_running": false, "took": 2000, "timed_out": false, "hits": ... }
另一种更轻量级的方法来检查异步搜索的进度是使用带有搜索 ID 的 获取异步 EQL 状态 API。
resp = client.eql.get_status( id="FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", ) print(resp)
response = client.eql.get_status( id: 'FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=' ) puts response
const response = await client.eql.getStatus({ id: "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", }); console.log(response);
GET /_eql/search/status/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=
{ "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", "is_running": false, "is_partial": false, "expiration_time_in_millis": 1611690295000, "completion_status": 200 }
更改搜索保留期
编辑默认情况下,EQL 搜索 API 会将异步搜索存储五天。在此期限之后,任何搜索及其结果都将被删除。使用 keep_alive
参数来更改此保留期。
resp = client.eql.search( index="my-data-stream", keep_alive="2d", wait_for_completion_timeout="2s", query="\n process where process.name == \"cmd.exe\"\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", keep_alive: "2d", wait_for_completion_timeout: "2s", query: '\n process where process.name == "cmd.exe"\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "keep_alive": "2d", "wait_for_completion_timeout": "2s", "query": """ process where process.name == "cmd.exe" """ }
您可以使用 获取异步 EQL 搜索 API 的 keep_alive
参数来稍后更改保留期。新的保留期将在获取请求运行后开始。
resp = client.eql.get( id="FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", keep_alive="5d", ) print(resp)
response = client.eql.get( id: 'FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=', keep_alive: '5d' ) puts response
const response = await client.eql.get({ id: "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", keep_alive: "5d", }); console.log(response);
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
在 keep_alive
期限结束之前,使用 删除异步 EQL 搜索 API 手动删除异步 EQL 搜索。如果搜索仍在进行中,Elasticsearch 将取消搜索请求。
resp = client.eql.delete( id="FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", ) print(resp)
response = client.eql.delete( id: 'FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=' ) puts response
const response = await client.eql.delete({ id: "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", }); console.log(response);
DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=
存储同步 EQL 搜索
编辑默认情况下,EQL 搜索 API 仅存储异步搜索。要保存同步搜索,请将 keep_on_completion
设置为 true
。
resp = client.eql.search( index="my-data-stream", keep_on_completion=True, wait_for_completion_timeout="2s", query="\n process where process.name == \"cmd.exe\"\n ", ) print(resp)
const response = await client.eql.search({ index: "my-data-stream", keep_on_completion: true, wait_for_completion_timeout: "2s", query: '\n process where process.name == "cmd.exe"\n ', }); console.log(response);
GET /my-data-stream/_eql/search { "keep_on_completion": true, "wait_for_completion_timeout": "2s", "query": """ process where process.name == "cmd.exe" """ }
响应包含一个搜索 ID。 is_partial
和 is_running
都为 false
,表示 EQL 搜索是同步的并返回了完整的结果。
{ "id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=", "is_partial": false, "is_running": false, "took": 52, "timed_out": false, "hits": ... }
使用 获取异步 EQL 搜索 API 以便稍后获取相同的结果。
resp = client.eql.get( id="FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=", ) print(resp)
response = client.eql.get( id: 'FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=' ) puts response
const response = await client.eql.get({ id: "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=", }); console.log(response);
GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=
保存的同步搜索仍然受 keep_alive
参数的保留期限制。当此期限结束时,搜索及其结果将被删除。
您也可以使用 获取异步 EQL 状态 API 来检查已保存的同步搜索的状态,而无需结果。
您还可以使用 删除异步 EQL 搜索 API 手动删除已保存的同步搜索。
跨集群运行 EQL 搜索
编辑此功能处于技术预览阶段,可能会在未来的版本中更改或删除。Elastic 将努力修复任何问题,但技术预览版中的功能不受官方 GA 功能的支持 SLA 约束。
EQL 搜索 API 支持 跨集群搜索。但是,如果本地和远程集群的版本低于 7.17.7(包含)或低于 8.5.1(包含),则它们必须使用相同的 Elasticsearch 版本。
以下 集群更新设置 请求添加了两个远程集群:cluster_one
和 cluster_two
。
resp = client.cluster.put_settings( persistent={ "cluster": { "remote": { "cluster_one": { "seeds": [ "127.0.0.1:9300" ] }, "cluster_two": { "seeds": [ "127.0.0.1:9301" ] } } } }, ) print(resp)
response = client.cluster.put_settings( body: { persistent: { cluster: { remote: { cluster_one: { seeds: [ '127.0.0.1:9300' ] }, cluster_two: { seeds: [ '127.0.0.1:9301' ] } } } } } ) puts response
const response = await client.cluster.putSettings({ persistent: { cluster: { remote: { cluster_one: { seeds: ["127.0.0.1:9300"], }, cluster_two: { seeds: ["127.0.0.1:9301"], }, }, }, }, }); console.log(response);
PUT /_cluster/settings { "persistent": { "cluster": { "remote": { "cluster_one": { "seeds": [ "127.0.0.1:9300" ] }, "cluster_two": { "seeds": [ "127.0.0.1:9301" ] } } } } }
要定位远程集群上的数据流或索引,请使用 <cluster>:<target>
语法。
resp = client.eql.search( index="cluster_one:my-data-stream,cluster_two:my-data-stream", query="\n process where process.name == \"regsvr32.exe\"\n ", ) print(resp)
const response = await client.eql.search({ index: "cluster_one:my-data-stream,cluster_two:my-data-stream", query: '\n process where process.name == "regsvr32.exe"\n ', }); console.log(response);
GET /cluster_one:my-data-stream,cluster_two:my-data-stream/_eql/search { "query": """ process where process.name == "regsvr32.exe" """ }
EQL 断路器设置
编辑相关的断路器设置可以在断路器页面中找到。