节流过滤器插件

如有任何关于插件的问题，请在 Discuss 论坛中发帖提问。如发现错误或有功能请求，请在 Github 中提交问题。如需查看 Elastic 支持的插件列表，请查阅 Elastic 支持矩阵。

节流过滤器用于限制事件数量。该过滤器配置了下限“before_count”和上限“after_count”，以及一个时间段。所有通过过滤器的事件都将根据其键和事件时间戳进行计数。只要计数小于“before_count”或大于“after_count”，事件就会被“节流”，这意味着过滤器将被视为成功，并且任何标签或字段都将被添加（或删除）。

该插件是线程安全的，并能正确跟踪过去的事件。

例如，如果您想节流事件，以便您只在发生 2 次事件后才接收一个事件，并且在 10 分钟内最多接收 3 个事件，则可以使用以下配置

    period => 600
    max_age => 1200
    before_count => 3
    after_count => 5

这将导致

event 1 - throttled (successful filter, period start)
event 2 - throttled (successful filter)
event 3 - not throttled
event 4 - not throttled
event 5 - not throttled
event 6 - throttled (successful filter)
event 7 - throttled (successful filter)
event x - throttled (successful filter)
period end
event 1 - throttled (successful filter, period start)
event 2 - throttled (successful filter)
event 3 - not throttled
event 4 - not throttled
event 5 - not throttled
event 6 - throttled (successful filter)
...

另一个例子是，如果您想节流事件，以便每小时只接收一个事件，则可以使用以下配置

    period => 3600
    max_age => 7200
    before_count => -1
    after_count => 1

这将导致

event 1 - not throttled (period start)
event 2 - throttled (successful filter)
event 3 - throttled (successful filter)
event 4 - throttled (successful filter)
event x - throttled (successful filter)
period end
event 1 - not throttled (period start)
event 2 - throttled (successful filter)
event 3 - throttled (successful filter)
event 4 - throttled (successful filter)
...

一个常见的用例是使用节流过滤器在 3 次之前和 5 次之后节流事件，同时使用多个字段作为键，然后使用 drop 过滤器删除节流事件。此配置可能如下所示：

    filter {
      throttle {
        before_count => 3
        after_count => 5
        period => 3600
        max_age => 7200
        key => "%{host}%{message}"
        add_tag => "throttled"
      }
      if "throttled" in [tags] {
        drop { }
      }
    }

另一种情况是存储所有事件，但只向非节流事件发送电子邮件，这样在系统错误的情况下，操作员的收件箱不会被电子邮件淹没。此配置可能如下所示：

    filter {
      throttle {
        before_count => 3
        after_count => 5
        period => 3600
        max_age => 7200
        key => "%{message}"
        add_tag => "throttled"
      }
    }
    output {
      if "throttled" not in [tags] {
        email {
          from => "[email protected]"
          subject => "Production System Alert"
          to => "[email protected]"
          via => "sendmail"
          body => "Alert on %{host} from path %{path}:\n\n%{message}"
          options => { "location" => "/usr/sbin/sendmail" }
        }
      }
      elasticsearch_http {
        host => "localhost"
        port => "19200"
      }
    }

接收到事件后，事件键将存储在 key_cache 中。该键引用一个 timeslot_cache。事件将根据事件的时间戳分配到一个时间段（动态创建）。时间段计数器递增。当在同一“时间段”内接收到下一个事件（相同的键）时，它将分配到相同的时间段。时间段计数器再次递增。

如果超过最大年龄，则时间段将过期。年龄是根据最新的事件时间戳和 max_age 配置选项计算的。

---[::.. DESIGN ..::]---

- [key_cache] - -- [timeslot_cache] -- | | | @created: 1439839636 | | @latest: 1439839836 | [a.b.c] ⇒ ---------------------- | [1439839636] ⇒ 1 | | [1439839736] ⇒ 3 | | [1439839836] ⇒ 2 | ----------------------

                   +-- [timeslot_cache] --+
                   | @created: eeeeeeeeee |
                   | @latest:  llllllllll |
   [x.y.z]  =>     +----------------------+
                   | [0000000060] => x    |
                   | [0000000120] => y    |
|               |  | [..........] => N    |
+---------------+  +----------------------+

Frank de Jong (@frapex) Mike Pilone (@mikepilone)

仅当大于当前值时才更新

此插件支持以下配置选项以及稍后描述的 常用选项。

另请参见 常用选项，了解所有过滤器插件支持的选项列表。

所有过滤器插件都支持这些配置选项

    filter {
      throttle {
        add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      }
    }

    # You can also add multiple fields at once:
    filter {
      throttle {
        add_field => {
          "foo_%{somefield}" => "Hello world, from %{host}"
          "new_field" => "new_static_value"
        }
      }
    }

    filter {
      throttle {
        add_tag => [ "foo_%{somefield}" ]
      }
    }

    # You can also add multiple tags at once:
    filter {
      throttle {
        add_tag => [ "foo_%{somefield}", "taggedy_tag"]
      }
    }

    filter {
      throttle {
        id => "ABC"
      }
    }

    filter {
      throttle {
        remove_field => [ "foo_%{somefield}" ]
      }
    }

    # You can also remove multiple fields at once:
    filter {
      throttle {
        remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
      }
    }

    filter {
      throttle {
        remove_tag => [ "foo_%{somefield}" ]
      }
    }

    # You can also remove multiple tags at once:
    filter {
      throttle {
        remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"]
      }
    }