节流过滤器插件

有关插件的问题，请在Discuss论坛中开一个主题。对于错误或功能请求，请在Github中开一个 issue。有关 Elastic 支持的插件列表，请查阅Elastic 支持矩阵。

节流过滤器用于限制事件的数量。该过滤器配置有下限“before_count”、上限“after_count”和时间段。所有通过过滤器的事件都将根据其键和事件时间戳进行计数。只要计数小于“before_count”或大于“after_count”，事件就会被“节流”，这意味着过滤器将被视为成功，并且将添加（或删除）任何标签或字段。

该插件是线程安全的，并且可以正确跟踪过去的事件。

例如，如果您想限制事件，以便在出现 2 次后仅接收一个事件，并且在 10 分钟内不超过 3 个事件，则可以使用以下配置

    period => 600
    max_age => 1200
    before_count => 3
    after_count => 5

这将导致

event 1 - throttled (successful filter, period start)
event 2 - throttled (successful filter)
event 3 - not throttled
event 4 - not throttled
event 5 - not throttled
event 6 - throttled (successful filter)
event 7 - throttled (successful filter)
event x - throttled (successful filter)
period end
event 1 - throttled (successful filter, period start)
event 2 - throttled (successful filter)
event 3 - not throttled
event 4 - not throttled
event 5 - not throttled
event 6 - throttled (successful filter)
...

另一个例子是，如果您想限制事件，以便每小时仅接收 1 个事件，则可以使用以下配置

    period => 3600
    max_age => 7200
    before_count => -1
    after_count => 1

这将导致

event 1 - not throttled (period start)
event 2 - throttled (successful filter)
event 3 - throttled (successful filter)
event 4 - throttled (successful filter)
event x - throttled (successful filter)
period end
event 1 - not throttled (period start)
event 2 - throttled (successful filter)
event 3 - throttled (successful filter)
event 4 - throttled (successful filter)
...

一个常见的用例是使用节流过滤器来限制 3 个之前的事件和 5 个之后的事件，同时使用多个字段作为键，然后使用 drop 过滤器删除节流事件。此配置可能显示为

    filter {
      throttle {
        before_count => 3
        after_count => 5
        period => 3600
        max_age => 7200
        key => "%{host}%{message}"
        add_tag => "throttled"
      }
      if "throttled" in [tags] {
        drop { }
      }
    }

另一种情况是存储所有事件，但仅发送非节流事件的电子邮件，这样在发生系统错误时，操作人员的收件箱就不会被电子邮件淹没。此配置可能显示为

    filter {
      throttle {
        before_count => 3
        after_count => 5
        period => 3600
        max_age => 7200
        key => "%{message}"
        add_tag => "throttled"
      }
    }
    output {
      if "throttled" not in [tags] {
        email {
          from => "[email protected]"
          subject => "Production System Alert"
          to => "[email protected]"
          via => "sendmail"
          body => "Alert on %{host} from path %{path}:\n\n%{message}"
          options => { "location" => "/usr/sbin/sendmail" }
        }
      }
      elasticsearch_http {
        host => "localhost"
        port => "19200"
      }
    }

当收到事件时，事件键存储在 key_cache 中。该键引用 timeslot_cache。根据事件的时间戳，事件被分配到一个时间槽（动态创建）。时间槽计数器递增。当在相同的“期间”内收到下一个事件（相同的键）时，它被分配到相同的时间槽。时间槽计数器再次递增。

如果超过了最大年龄，时间槽将过期。年龄根据最新的事件时间戳和 max_age 配置选项计算。

---[::.. DESIGN ..::]---

- [key_cache] - -- [timeslot_cache] -- | | | @created: 1439839636 | | @latest: 1439839836 | [a.b.c] ⇒ ---------------------- | [1439839636] ⇒ 1 | | [1439839736] ⇒ 3 | | [1439839836] ⇒ 2 | ----------------------

                   +-- [timeslot_cache] --+
                   | @created: eeeeeeeeee |
                   | @latest:  llllllllll |
   [x.y.z]  =>     +----------------------+
                   | [0000000060] => x    |
                   | [0000000120] => y    |
|               |  | [..........] => N    |
+---------------+  +----------------------+

Frank de Jong (@frapex) Mike Pilone (@mikepilone)

仅当大于当前值时才更新

此插件支持以下配置选项以及稍后描述的通用选项。

另请参阅通用选项，以获取所有过滤器插件支持的选项列表。

所有过滤器插件都支持这些配置选项

    filter {
      throttle {
        add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      }
    }

    # You can also add multiple fields at once:
    filter {
      throttle {
        add_field => {
          "foo_%{somefield}" => "Hello world, from %{host}"
          "new_field" => "new_static_value"
        }
      }
    }

    filter {
      throttle {
        add_tag => [ "foo_%{somefield}" ]
      }
    }

    # You can also add multiple tags at once:
    filter {
      throttle {
        add_tag => [ "foo_%{somefield}", "taggedy_tag"]
      }
    }

    filter {
      throttle {
        id => "ABC"
      }
    }

    filter {
      throttle {
        remove_field => [ "foo_%{somefield}" ]
      }
    }

    # You can also remove multiple fields at once:
    filter {
      throttle {
        remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
      }
    }

    filter {
      throttle {
        remove_tag => [ "foo_%{somefield}" ]
      }
    }

    # You can also remove multiple tags at once:
    filter {
      throttle {
        remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"]
      }
    }