Filters

Reference documentation

What are Sensu filters?

Sensu Filters (also called Event Filters) allow you to filter events destined for one or more event Handlers. Sensu filters inspect event data and match its keys/values with filter definition attributes, to determine if the event should be passed to an event handler. Filters are commonly used to filter recurring events (i.e. to eliminate notification noise) and to filter events from systems in pre-production environments.

When to use a filter

Sensu Filters allow you to configure conditional logic to be applied during the event processing flow. Compared to executing an event handler, evaluating event filters is an inexpensive operation which can provide overall monitoring performance gains by reducing the number of events that need to be handled. Additionally, by using Sensu Filters, instead of building conditional logic into custom Handlers, conditional logic can be applied to multiple Handlers, and monitoring configuration stays DRY.

How do Sensu filters work?

Sensu Filters are applied when Event Handlers are configured to use one or more Filters. Prior to executing a Handler, the Sensu server will apply any Filters configured for the Handler to the Event Data. If the Event is not removed by the Filter(s) (i.e. filtered out), the Handler will be executed. The filter analysis flow performs these steps:

  • When the Sensu server is processing an Event, it will check for the definition of a handler (or handlers). Prior to executing each Handler, the Sensu server will first apply any configured filter (or filters) for the Handler
  • If multiple filters are configured for a Handler, they are executed sequentially
  • Filter attributes are compared with Event data
  • Filters can be inclusive (only matching events are handled) or exclusive (matching events are not handled)
  • As soon as a Filter removes an Event (i.e. filters it out), no further analysis is performed and the Event Handler will not be executed

NOTE: Filters specified in a handler set definition have no effect. Filters must be specified in individual handler definitions.

Inclusive and Exclusive Filtering

Filters can be inclusive ("negate": false) or exclusive ("negate": true). Configuring a handler to use multiple inclusive filters is the equivalent of using an AND query operator (i.e. only handle events if they match inclusive filters x AND y AND z). Configuring a handler to use multiple exclusive filters is the equivalent of using an OR operator (i.e. only handle events if they don’t match x OR y OR z).

  • Inclusive filtering: by setting the filter definition attribute "negate": false, only events that match the defined filter attributes are handled.

  • Exclusive filtering: by setting the filter definition attribute "negate": true, events are only handled if they do not match the defined filter attributes.

NOTE: unless otherwise configured in the filter definition, the default filtering behavior is inclusive filtering (i.e. "negate": false).

Filter attribute comparison

Filter attributes are compared directly with their event data counterparts. For inclusive filter definitions (i.e. "negate": false), matching attributes will result in the filter returning a true value; for exclusive filter definitions (i.e. "negate": true), matching attributes will result in the filter returning a false value (i.e. the event does not pass through the filter). Filters that return a true value will continue to be processed — via additional filters (if defined), mutators (if defined), and handlers.

EXAMPLE

The following example filter definition, entitled production_filter will match event data with a custom client definition attribute "environment": "production".

{
  "filters": {
    "production_filter": {
      "negate": false,
      "attributes": {
        "client": {
          "environment": "production"
        }
      }
    }
  }
}

Filter attribute evaluation

When more complex conditional logic is needed than direct filter attribute comparison, Sensu filters provide support for attribute evaluation using Ruby expressions. When a Filter attribute value is a string beginning with eval:, the remainder is evaluated as a Ruby expression. The Ruby expression is evaluated in a “sandbox” and provided a single variable (value) which is equal to the event data attribute value being compared. If the evaluated expression returns true, the attribute is a match.

Example: Handling state change only

Some teams migrating to Sensu have asked about reproducing the behavior of their old monitoring system which alerts only on state change. This state_change_only inclusive filter provides such.

{
  "filters": {
    "state_change_only": {
      "negate": false,
      "attributes": {
        "occurrences": "eval: value == 1 || ':::action:::' == 'resolve'"
      }
    }
  }
}

This eval filter is effective because value of event occurrences is reset on each state change, except when the event action is :resolve. The resolve action is set on an event when its check result status is 0 following one or more prior non-zero statuses.

Example: Handling repeated events

The following example filter definition, entitled filter_interval_60_hourly, will match event data with a check interval of 60 seconds, and an occurrences value of 1 (i.e. the first occurrence) -OR- any occurrences value that is evenly divisible by 60 (via a modulo operator calculation; i.e. calculating the remainder after dividing occurrences by 60).

Note that negate is true, making this an exclusive filter; if evaluation returns false, the event will be handled.

{
  "filters": {
    "filter_interval_60_hourly": {
      "negate": true,
      "attributes": {
        "check": {
          "interval": 60
        },
        "occurrences": "eval: value != 1 && value % 60 != 0"
      }
    }
  }
}

The next example will apply the same logic as the previous example, but for checks with a 30 second interval.

{
  "filters": {
    "filter_interval_30_hourly": {
      "negate": true,
      "attributes": {
        "check": {
          "interval": 30
        },
        "occurrences": "eval: value != 1 && value % 120 != 0"
      }
    }
  }
}

NOTE: The effect of both of these filters is that they will only allow an events with 30-second or 60-second intervals to be handled on the first occurrence of the event, and again every hour. Previous examples in the older Sensu docs have not included the "check": { "interval": 60 } attribute, which has confused some users because filtering based on occurrences alone assumes some understanding of the relationship between occurrences and interval, which isn’t always obvious.

Example: Handling events during “office hours” only

This filter evaluates the event timestamp to determine if the event occurred between 9 AM and 5 PM on a weekday. Remember that negate defaults to false, so this is an inclusive filter. If evaluation returns false, the event will not be

{
  "filters": {
    "nine_to_fiver": {
      "negate": false,
      "attributes": {
        "timestamp": "eval: [1,2,3,4,5].include?(Time.at(value).wday) && Time.at(value).hour.between?(9,17)"
      }
    }
  }
}

Filter attribute eval tokens

What are filter attribute eval tokens?

Sensu filters attributes may be evaluated using Ruby expressions, which evaluations provide support for comparing a single event attribute variable against a basic logical statement (e.g. is value greater than 60?). When additional variables are needed beyond the single value variable provided by eval:, tokens may be used. Eval tokens are filter attribute placeholders that can be replaced by Sensu check definition attributes and client definition attributes (including custom attributes).

Example filter attribute eval token

The following is an example Sensu filter definition, which is using a token (:::check.occurrences|60:::) as a secondary attribute in the Ruby eval expression. The token will be replaced by the check definition attribute named occurrences if it is defined, otherwise it will use the fallback value of 60.

{
  "filters": {
    "occurrences": {
      "negate": true,
      "attributes": {
        "occurrences": "eval: value > :::check.occurrences|60:::"
      }
    }
  }
}

This example would be useful for filtering events that don’t exceed a minimum number of occurrences as configured in the check definition.

Filter attribute eval token specification

Eval token substitution syntax

Eval tokens are invoked by wrapping event data attributes with “triple colons” (i.e. three colon characters before and after the attribute, i.e. :::). Nested event data attributes may be accessed via “dot notation” (e.g. check.occurrences)

Eval token default values

Eval token default values can be used as a fallback in the event that an eval token attribute is not satisfied by event data. Eval token default values are separated by a pipe character (|), and can be used to provide a “fallback value” for events that are missing the declared token attribute.

  • :::check.occurrences|60::: would be replaced with a check definition attribute called occurrences. If occurrences is not defined in the check definition, the default (or fallback) value of 60 will be used.

NOTE: if an eval token default value is not provided (i.e. as a fallback value), and the event data does not contain a matching eval token attribute, an log entry indicating an error called "filter eval unmatched tokens" will be published to the Sensu server log.

Built-in Filters

Occurrences filter

The occurrences filter is included in every installation of Sensu. It lets you control the number of duplicate events that reach the handler. You can apply the occurrences filter to a handler using the filters handler definition attribute.

Here’s an example of a handler definition that uses the occurrences filter:

{
  "handlers": {
    "email": {
      "type": "pipe",
      "command": "email.rb",
      "filters": ["occurrences"]
    }
  }
}

Occurrences filter attributes

The occurrences filter lets you configure the number of occurrences and the refresh interval at the check level using two check definition attributes: occurrences and refresh.

Here’s an example of a check definition that passes events to the email handler starting with the second occurrence every 60 minutes:

{
  "checks": {
    "check-http": {
      "command": "check-http.rb -u https://localhost:8080/api/v1/health",
      "subscribers": ["web_application"],
      "interval": 20,
      "handlers": ["email"],
      "occurrences": 2,
      "refresh": 3600
    }
  }
}
occurrences
description The number of events that must occur before an event is handled for the check
required True if the specified handler uses the occurrences filter
type Integer
default 1
example
"occurrences": 2
refresh
description Time in seconds until event occurrences are handled for the check again
required False
type Integer
default 1800
example
"refresh": 3600

Check dependencies filter

The check dependencies filter is included in every installation of Sensu. It lets you specify checks that are dependencies of a given check, so if the dependent check is already alerting, Sensu won’t handle the check that is alerting as a result of its dependency. This lets you reduce notification noise by only alerting for the root cause of a given failure. The check dependencies filter can be applied to a handler using the filters handler definition attribute.

Here’s an example of a handler definition that uses the checks dependencies filter:

{
  "handlers": {
    "custom_mailer": {
      "type": "pipe",
      "command": "custom_mailer.rb",
      "filters": ["check_dependencies"]
    }
  }
}

Check dependencies filter attributes

The check dependencies filter uses a custom check definition attribute: dependencies. The dependencies attribute should define an array containing names of checks, client/check pairs, or subscription/check pairs.

Here’s an example of a check definition that will be filtered if a check named mysql from the same client is already alerting:

{
  "checks": {
    "web_application_api": {
      "command": "check-http.rb -u https://localhost:8080/api/v1/health",
      "subscribers": ["web_application"],
      "interval": 20,
      "dependencies": ["mysql"]
    }
  }
}

You can also define a dependency by specifying the client and the check. Here’s an example of a check definition that will be filtered if a check named mysql from client db-01 is already alerting:

{
  "checks": {
    "web_application_api": {
      "command": "check-http.rb -u https://localhost:8080/api/v1/health",
      "subscribers": ["web_application"],
      "interval": 20,
      "dependencies": ["db-01/mysql"]
    }
  }
}

Or, define a dependency by specifying the subscription and the check. Here’s an example of a check definition that will be filtered if a check named mysql is already alerting on any client subscribed to the db-nodes subscription:

{
  "checks": {
    "web_application_api": {
      "command": "check-http.rb -u https://localhost:8080/api/v1/health",
      "subscribers": ["web_application"],
      "interval": 20,
      "dependencies": ["subscription:db-nodes/mysql"]
    }
  }
}

WARNING: Specifying a subscription/check pair in the check dependencies filter may impact Sensu’s performance in cases where the events API regularly returns thousands of events.

dependencies
description An array containing names of checks, client/check pairs, or subscription/check pairs. Subscription/check pairs must be in the format subscription:subscription-name/check-name.
required True if specified handler uses the check dependencies filter
type Array
example
"dependencies": ["db-01/mysql", "apache", "subscription:db-nodes/mysql"]

Filter configuration

Example filter definition

The following is an example Sensu filter definition, a JSON configuration file located at /etc/sensu/conf.d/filter_production.json. This is an inclusive filter definition called production. The effect of this filter is that only events with the custom client attribute "environment": "production" will be handled.

{
  "filters": {
    "production": {
      "attributes": {
        "client": {
          "environment": "production"
        }
      },
      "negate": false
    }
  }
}

Filter definition specification

Filter naming

Each filter definition has a unique name, used for the definition key. Every filter definition is within the "filters": {} definition scope.

  • A unique string used to name/identify the filter
  • Cannot contain special characters or spaces
  • Validated with Ruby regex /^[\w\.-]+$/.match("filter-name")

Filter attributes

negate
description If the filter will negate events that match the filter attributes.NOTE: see Inclusive and exclusive filtering for more information.
required false
type Boolean
default false
example
"negate": true
attributes
description Filter attributes to be compared with Event data.
required true
type Hash
example
"attributes": {
  "check": {
    "team": "ops"
  }
}
when
description The when definition scope, used to determine when a filter is applied (time windows).
required false
type Hash
example
"when": {
  "days": {
    "all": [
      {
        "begin": "5:00 PM",
        "end": "8:00 AM"
      }
    ]
  }
}

when attributes

The following attributes are configured within the {"filters": { "FILTER": { "when": {} } } } configuration scope (where FILTER is a valid filter name).

EXAMPLE
{
  "filters": {
    "offhours": {
      "attributes": {
        "client": {
          "environment": "production"
        }
      },
      "when": {
        "days": {
          "all": [
            {
              "begin": "5:00 PM",
              "end": "8:00 AM"
            }
          ]
        }
      }
    }
  }
}
ATTRIBUTES
days
description A hash of days of the week or ‘all’, each day specified defines one or more time windows in which the filter is applied.
required false (unless when is configured)
type Hash
example
"days": {
  "all": [
    {
      "begin": "5:00 PM",
      "end": "8:00 AM"
    }
  ],
  "friday": [
    {
      "begin": "12:00 PM",
      "end": "5:00 PM"
    }
  ]
}