节流过滤器插件

有关插件的问题，请在 Discuss 论坛中发布主题。对于错误或功能请求，请在 Github 中打开问题。有关 Elastic 支持的插件列表，请参阅 Elastic 支持矩阵。

节流过滤器用于限制事件数量。过滤器配置了“before_count”的下限和“after_count”的上限，以及一段时间。所有通过过滤器的事件将根据其键和事件时间戳进行计数。只要计数小于“before_count”或大于“after_count”，事件将被“节流”，这意味着过滤器将被视为成功，并且将添加（或删除）任何标签或字段。

该插件是线程安全的，可以正确跟踪过去的事件。

例如，如果您想限制事件，以便您只在 2 次出现后收到一个事件，并且在 10 分钟内最多收到 3 个事件，您将使用以下配置

    period => 600
    max_age => 1200
    before_count => 3
    after_count => 5

这将导致

event 1 - throttled (successful filter, period start)
event 2 - throttled (successful filter)
event 3 - not throttled
event 4 - not throttled
event 5 - not throttled
event 6 - throttled (successful filter)
event 7 - throttled (successful filter)
event x - throttled (successful filter)
period end
event 1 - throttled (successful filter, period start)
event 2 - throttled (successful filter)
event 3 - not throttled
event 4 - not throttled
event 5 - not throttled
event 6 - throttled (successful filter)
...

另一个示例是，如果您想限制事件，以便您每小时只收到 1 个事件，您将使用以下配置

    period => 3600
    max_age => 7200
    before_count => -1
    after_count => 1

这将导致

event 1 - not throttled (period start)
event 2 - throttled (successful filter)
event 3 - throttled (successful filter)
event 4 - throttled (successful filter)
event x - throttled (successful filter)
period end
event 1 - not throttled (period start)
event 2 - throttled (successful filter)
event 3 - throttled (successful filter)
event 4 - throttled (successful filter)
...

一个常见的用例是使用节流过滤器在 3 次之前和 5 次之后限制事件，同时使用多个字段作为键，然后使用 drop 过滤器删除节流事件。此配置可能如下所示

    filter {
      throttle {
        before_count => 3
        after_count => 5
        period => 3600
        max_age => 7200
        key => "%{host}%{message}"
        add_tag => "throttled"
      }
      if "throttled" in [tags] {
        drop { }
      }
    }

另一种情况是存储所有事件，但只向非节流事件发送电子邮件，这样在系统错误的情况下，操作人员的收件箱不会被电子邮件淹没。此配置可能如下所示

    filter {
      throttle {
        before_count => 3
        after_count => 5
        period => 3600
        max_age => 7200
        key => "%{message}"
        add_tag => "throttled"
      }
    }
    output {
      if "throttled" not in [tags] {
        email {
          from => "[email protected]"
          subject => "Production System Alert"
          to => "[email protected]"
          via => "sendmail"
          body => "Alert on %{host} from path %{path}:\n\n%{message}"
          options => { "location" => "/usr/sbin/sendmail" }
        }
      }
      elasticsearch_http {
        host => "localhost"
        port => "19200"
      }
    }

当收到事件时，事件键将存储在 key_cache 中。该键引用 timeslot_cache。事件将根据事件的时间戳分配到一个时间段（动态创建）。时间段计数器递增。当收到下一个事件（相同的键）时，在同一“周期”内，它将分配到相同的时间段。时间段计数器再次递增。

如果超过最大年龄，时间段将过期。年龄是根据最新事件时间戳和 max_age 配置选项计算的。

---[::.. DESIGN ..::]---

- [key_cache] - -- [timeslot_cache] -- | | | @created: 1439839636 | | @latest: 1439839836 | [a.b.c] ⇒ ---------------------- | [1439839636] ⇒ 1 | | [1439839736] ⇒ 3 | | [1439839836] ⇒ 2 | ----------------------

                   +-- [timeslot_cache] --+
                   | @created: eeeeeeeeee |
                   | @latest:  llllllllll |
   [x.y.z]  =>     +----------------------+
                   | [0000000060] => x    |
                   | [0000000120] => y    |
|               |  | [..........] => N    |
+---------------+  +----------------------+

Frank de Jong (@frapex) Mike Pilone (@mikepilone)

仅在大于当前值时更新

此插件支持以下配置选项，以及后面描述的 通用选项。

另请参阅 通用选项，了解所有过滤器插件支持的选项列表。

以下配置选项由所有过滤器插件支持

    filter {
      throttle {
        add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      }
    }

    # You can also add multiple fields at once:
    filter {
      throttle {
        add_field => {
          "foo_%{somefield}" => "Hello world, from %{host}"
          "new_field" => "new_static_value"
        }
      }
    }

    filter {
      throttle {
        add_tag => [ "foo_%{somefield}" ]
      }
    }

    # You can also add multiple tags at once:
    filter {
      throttle {
        add_tag => [ "foo_%{somefield}", "taggedy_tag"]
      }
    }

    filter {
      throttle {
        id => "ABC"
      }
    }

    filter {
      throttle {
        remove_field => [ "foo_%{somefield}" ]
      }
    }

    # You can also remove multiple fields at once:
    filter {
      throttle {
        remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
      }
    }

    filter {
      throttle {
        remove_tag => [ "foo_%{somefield}" ]
      }
    }

    # You can also remove multiple tags at once:
    filter {
      throttle {
        remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"]
      }
    }