Csv 编解码器插件

编辑
  • 插件版本:v1.1.0
  • 发布日期:2021-07-28
  • 变更日志

其他版本请参见 版本化插件文档

安装

编辑

对于默认情况下未捆绑的插件,可以通过运行 bin/logstash-plugin install logstash-codec-csv 轻松安装。更多详情请参见 使用插件

获取帮助

编辑

如有关于插件的问题,请在 Discuss 论坛中发帖提问。对于错误或功能请求,请在 Github 中提交 issue。如需查看 Elastic 支持的插件列表,请查阅 Elastic 支持矩阵

描述

编辑

csv 编解码器接收 CSV 数据,对其进行解析并传递。

与 Elastic 通用架构 (ECS) 的兼容性

编辑

除了在启用 ECS 且未设置 target 时发出警告外,该插件的行为相同。

设置 target 选项可避免潜在的架构冲突。

Csv 编解码器配置选项

编辑
设置 输入类型 必需

autodetect_column_names

布尔值

autogenerate_column_names

布尔值

字符集

字符串,其中之一为 ["ASCII-8BIT", "UTF-8", "US-ASCII", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-JP", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "Windows-31J", "Windows-1250", "Windows-1251", "Windows-1252", "IBM437", "IBM737", "IBM775", "CP850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "IBM860", "IBM861", "IBM862", "IBM863", "IBM864", "IBM865", "IBM866", "IBM869", "Windows-1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "CP951", "IBM037", "stateless-ISO-2022-JP", "eucJP-ms", "CP51932", "EUC-JIS-2004", "GB12345", "ISO-2022-JP", "ISO-2022-JP-2", "CP50220", "CP50221", "Windows-1256", "Windows-1253", "Windows-1255", "Windows-1254", "TIS-620", "Windows-874", "Windows-1257", "MacJapanese", "UTF-7", "UTF8-MAC", "UTF-16", "UTF-32", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "BINARY", "CP437", "CP737", "CP775", "IBM850", "CP857", "CP860", "CP861", "CP862", "CP863", "CP864", "CP865", "CP866", "CP869", "CP1258", "Big5-HKSCS:2008", "ebcdic-cp-us", "eucJP", "euc-jp-ms", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "CP936", "ISO2022-JP", "ISO2022-JP2", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "CP1256", "ISO8859-7", "CP1253", "ISO8859-8", "CP1255", "ISO8859-9", "CP1254", "ISO8859-10", "ISO8859-11", "CP874", "ISO8859-13", "CP1257", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "external", "locale"]

数组

转换

哈希表

ecs_compatibility

字符串

include_headers

布尔值

quote_char

字符串

分隔符

字符串

skip_empty_columns

布尔值

目标

字符串

 

autodetect_column_names

编辑

定义是否应从标题列自动检测列名。默认为 false。

autogenerate_column_names

编辑

定义是否应自动生成列名。默认为 true。如果设置为 false,则不会解析没有指定标题的列。

charset

编辑
  • 值可以是:ASCII-8BIT, UTF-8, US-ASCII, Big5, Big5-HKSCS, Big5-UAO, CP949, Emacs-Mule, EUC-JP, EUC-KR, EUC-TW, GB2312, GB18030, GBK, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14, ISO-8859-15, ISO-8859-16, KOI8-R, KOI8-U, Shift_JIS, UTF-16BE, UTF-16LE, UTF-32BE, UTF-32LE, Windows-31J, Windows-1250, Windows-1251, Windows-1252, IBM437, IBM737, IBM775, CP850, IBM852, CP852, IBM855, CP855, IBM857, IBM860, IBM861, IBM862, IBM863, IBM864, IBM865, IBM866, IBM869, Windows-1258, GB1988, macCentEuro, macCroatian, macCyrillic, macGreek, macIceland, macRoman, macRomania, macThai, macTurkish, macUkraine, CP950, CP951, IBM037, stateless-ISO-2022-JP, eucJP-ms, CP51932, EUC-JIS-2004, GB12345, ISO-2022-JP, ISO-2022-JP-2, CP50220, CP50221, Windows-1256, Windows-1253, Windows-1255, Windows-1254, TIS-620, Windows-874, Windows-1257, MacJapanese, UTF-7, UTF8-MAC, UTF-16, UTF-32, UTF8-DoCoMo, SJIS-DoCoMo, UTF8-KDDI, SJIS-KDDI, ISO-2022-JP-KDDI, stateless-ISO-2022-JP-KDDI, UTF8-SoftBank, SJIS-SoftBank, BINARY, CP437, CP737, CP775, IBM850, CP857, CP860, CP861, CP862, CP863, CP864, CP865, CP866, CP869, CP1258, Big5-HKSCS:2008, ebcdic-cp-us, eucJP, euc-jp-ms, EUC-JISX0213, eucKR, eucTW, EUC-CN, eucCN, CP936, ISO2022-JP, ISO2022-JP2, ISO8859-1, ISO8859-2, ISO8859-3, ISO8859-4, ISO8859-5, ISO8859-6, CP1256, ISO8859-7, CP1253, ISO8859-8, CP1255, ISO8859-9, CP1254, ISO8859-10, ISO8859-11, CP874, ISO8859-13, CP1257, ISO8859-14, ISO8859-15, ISO8859-16, CP878, MacJapan, ASCII, ANSI_X3.4-1968, 646, CP65000, CP65001, UTF-8-MAC, UTF-8-HFS, UCS-2BE, UCS-4BE, UCS-4LE, CP932, csWindows31J, SJIS, PCK, CP1250, CP1251, CP1252, external, locale
  • 默认值为 "UTF-8"

用于 convert 选项的有效转换类型列表 此编解码器中使用的字符编码。例如,“UTF-8”和“CP1252”。

columns

编辑
  • 值类型为 数组
  • 默认值为 []

解码时: 定义列名列表(按其在 CSV 中出现的顺序,就像标题行一样)。如果未配置 columns,或者未指定足够的列,则默认列名为“column1”、“column2”等。

编码时: 要包含在编码 CSV 中的字段名列表,按列出的顺序排列。

convert

编辑

定义要应用于列的一组数据类型转换。可能的转换包括:integerfloatdatedate_timeboolean

示例

    filter {
      csv {
        convert => { "column1" => "integer", "column2" => "boolean" }
      }
    }

ecs_compatibility

编辑
  • 值类型为 字符串
  • 支持的值为

    • disabled:在根级别添加 CSV 数据
    • v1v8:符合 Elastic 通用架构的行为(还会添加 [event][original]
  • 默认值取决于运行 Logstash 的版本

    • 当 Logstash 提供 pipeline.ecs_compatibility 设置时,其值将用作默认值
    • 否则,默认值为 disabled

控制此插件与 Elastic 通用架构 (ECS) 的兼容性。

include_headers

编辑

在输出插件中编码时,在每个编解码器生命周期中(不是每个事件)仅包含一次编码 CSV 中的标题。默认 ⇒ false

quote_char

编辑

定义用于引用 CSV 字段的字符。如果未指定,则默认为双引号 "。可选。

separator

编辑

定义列分隔符值。如果未指定,则默认为逗号 ,。可选。

skip_empty_columns

编辑

定义是否应跳过空列。默认为 false。如果设置为 true,则不包含任何值的列将不会被包含。

target

编辑
  • 值类型为 字符串
  • 此设置没有默认值。

定义用于放置行值的目標字段。如果未设置此设置,则 CSV 数据将存储在事件的根(顶级)位置。

例如,如果希望数据放在 document 字段下

    input {
      file {
        codec => csv {
          autodetect_column_names => true
          target => "[document]"
        }
      }
    }