Handlers reference
Sensu executes handlers during the process stage of the observability pipeline.
Handlers are actions the Sensu backend executes on events.
Several types of handlers are available.
The most common are pipe
handlers, which work similarly to checks and enable Sensu to interact with almost any computer program via standard streams.
- Pipe handlers send observation data (events) into arbitrary commands via stdin
- TCP/UDP handlers send observation data (events) to a remote socket
- Handler sets group event handlers and streamline groups of actions to execute for certain types of events (also called “set handlers”)
The handler stack concept describes a group of handlers or a handler set that escalates events through a series of different handlers.
Discover, download, and share Sensu handler dynamic runtime assets using Bonsai, the Sensu asset hub. Read Use dynamic runtime assets to install plugins to get started.
Pipe handlers
Pipe handlers are external commands that can consume event data via stdin.
Pipe handler example
This example shows a pipe handler resource definition with the minimum required attributes:
---
type: Handler
api_version: core/v2
metadata:
name: pipe_handler_minimum
spec:
command: command-example
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "pipe_handler_minimum"
},
"spec": {
"command": "command-example",
"type": "pipe"
}
}
Pipe handler command
Pipe handler definitions include a command
attribute, which is a command for the Sensu backend to execute.
Pipe handler command arguments
Pipe handler command
attributes may include command line arguments for controlling the behavior of the command
executable.
TCP/UDP handlers
TCP and UDP handlers enable Sensu to forward event data to arbitrary TCP or UDP sockets for external services to consume.
TCP/UDP handler example
This handler will send event data to a TCP socket (10.0.1.99:4444) and timeout if an acknowledgement (ACK
) is not received within 30 seconds:
---
type: Handler
api_version: core/v2
metadata:
name: tcp_handler
spec:
socket:
host: 10.0.1.99
port: 4444
type: tcp
timeout: 30
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "tcp_handler"
},
"spec": {
"type": "tcp",
"timeout": 30,
"socket": {
"host": "10.0.1.99",
"port": 4444
}
}
}
Change the type
from tcp
to udp
to configure a UDP handler:
---
type: Handler
api_version: core/v2
metadata:
name: udp_handler
spec:
socket:
host: 10.0.1.99
port: 4444
type: udp
timeout: 30
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "udp_handler"
},
"spec": {
"type": "udp",
"timeout": 30,
"socket": {
"host": "10.0.1.99",
"port": 4444
}
}
}
Handler sets
NOTE: We recommend using pipelines to configure multiple workflows for different handlers instead of handler sets.
Handler set definitions allow you to use a single named handler set to refer to groups of handlers. The handler set becomes a collection of individual actions to take (via each included handler) on event data.
For example, suppose you have already created these two handlers:
elasticsearch
to send all observation data to Elasticsearch.opsgenie
to send non-OK status alerts to your OpsGenie notification channel.
You can list both of these handlers in a handler set to automate and streamline your workflow, specifying type: set
:
---
type: Handler
api_version: core/v2
metadata:
name: send_events_notify_operator
spec:
handlers:
- elasticsearch
- opsgenie
type: set
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "send_events_notify_operator"
},
"spec": {
"type": "set",
"handlers": [
"elasticsearch",
"opsgenie"
]
}
}
Now you can route observation data to Elasticsearch and alerts to OpsGenie with a single handler definition, the send_events_notify_operator
handler set.
NOTE: Attributes defined in handler sets do not apply to the handlers they include.
For example, filters
and mutator
attributes defined in a handler set will have no effect on handlers.
Define these attributes in individual handlers instead, or use pipelines.
Handler stacks
NOTE: We recommend using pipelines to configure multiple workflows for escalating events through a series of handlers instead of handler stacks.
The handler stack concept refers to a group of handlers or a handler set that escalates events through a series of different handlers. For example, suppose you want a handler stack with three levels of escalation:
- Level 1: On the first occurrence, attempt remediation.
- Level 2: On the fifth occurrence, send an alert to Slack.
- Level 3: On the tenth occurrence, send an alert to PagerDuty. Continue to send this alert on every tenth occurrence thereafter until the incident is resolved.
A handler stack for this scenario requires three handlers to take the desired actions based on three corresponding event filters that control the escalation levels:
- Level 1 requires an event filter with the built-in
is_incident
filter plus an occurrence-based filter that uses an expression likeevent.check.occurrences ==1
and a corresponding remediation handler. - Level 2 requires an event filter with
is_incident
plus an occurrence-based filter that uses an expression likeevent.check.occurrences == 5
and a corresponding Slack handler. - Level 3 requires an event filter with
is_incident
plus an occurrence-based filter that uses an expression likeevent.check.occurrences % 10 == 0
to match event data with an occurrences value that is evenly divisible by 10 via a modulo operator calculation and a corresponding PagerDuty handler.
With these event filters and handlers configured, you can create a handler set that includes the three handlers in your stack. You can also list the three handlers in the handlers array in your check definition instead.
PRO TIP: This scenario relies on six different resources, three event filters and three handlers, to describe the handler stack concept, but you can use Sensu dynamic runtime assets and integrations to achieve the same escalating alert levels in other ways.
For example, you can use the is_incident
event filter in conjunction with the sensu/sensu-go-fatigue-check-filter asset to control event escalation.
The sensu/sensu-ansible-handler, sensu/sensu-rundeck-handler, and sensu/sensu-saltstack-handler auto-remediation integrations and the sensu/sensu-remediation-handler asset also include built-in occurrence- and severity-based event filtering.
Keepalive event handlers
Sensu keepalives are the heartbeat mechanism used to ensure that all registered Sensu agents are operational and can reach the Sensu backend.
You can connect keepalive events to your monitoring workflows using a keepalive handler.
Sensu looks for an event handler named keepalive
and automatically uses it to process keepalive events.
Suppose you want to receive Slack notifications for keepalive alerts, and you already have a Slack handler set up to process events.
To process keepalive events using the Slack pipeline, create a handler set named keepalive
and add the slack
handler to the handlers
array.
The resulting keepalive
handler set configuration will look like this example:
---
type: Handler
api_version: core/v2
metadata:
name: keepalive
spec:
handlers:
- slack
type: set
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "keepalive"
},
"spec": {
"type": "set",
"handlers": [
"slack"
]
}
}
You can also use the keepalive-handlers
configuration option to send keepalive events to any handler you have configured.
If you do not specify a keepalive handler with the keepalive-handlers
configuration option, the Sensu backend will use the default keepalive
handler and create an event in sensuctl and the Sensu web UI.
Handler specification
Top-level attributes
api_version | |
---|---|
description | Top-level attribute that specifies the Sensu API group and version. For handlers in this version of Sensu, the api_version should always be core/v2 . |
required | Required for handler definitions in wrapped-json or yaml format for use with sensuctl create . |
type | String |
example |
|
metadata | |
---|---|
description | Top-level collection of metadata about the handler that includes name , namespace , and created_by as well as custom labels and annotations . The metadata map is always at the top level of the handler definition. This means that in wrapped-json and yaml formats, the metadata scope occurs outside the spec scope. Read metadata attributes for details. |
required | Required for handler definitions in wrapped-json or yaml format for use with sensuctl create . |
type | Map of key-value pairs |
example |
|
spec | |
---|---|
description | Top-level map that includes the handler spec attributes. |
required | Required for handler definitions in wrapped-json or yaml format for use with sensuctl create . |
type | Map of key-value pairs |
example |
|
type | |
---|---|
description | Top-level attribute that specifies the sensuctl create resource type. Handlers should always be type Handler . |
required | Required for handler definitions in wrapped-json or yaml format for use with sensuctl create . |
type | String |
example |
|
Metadata attributes
annotations | |
---|---|
description | Non-identifying metadata to include with observation event data that you can access with event filters. You can use annotations to add data that’s meaningful to people or external tools that interact with Sensu. In contrast to labels, you cannot use annotations in API response filtering, sensuctl response filtering, or web UI views. |
required | false |
type | Map of key-value pairs. Keys and values can be any valid UTF-8 string. |
default | null |
example |
|
created_by | |
---|---|
description | Username of the Sensu user who created the handler or last updated the handler. Sensu automatically populates the created_by field when the handler is created or updated. |
required | false |
type | String |
example |
|
labels | |
---|---|
description | Custom attributes to include with observation event data that you can use for response and web UI view filtering. If you include labels in your event data, you can filter API responses, sensuctl responses, and web UI views based on them. In other words, labels allow you to create meaningful groupings for your data. Limit labels to metadata you need to use for response filtering. For complex, non-identifying metadata that you will not need to use in response filtering, use annotations rather than labels. |
required | false |
type | Map of key-value pairs. Keys can contain only letters, numbers, and underscores and must start with a letter. Values can be any valid UTF-8 string. |
default | null |
example |
|
name | |
---|---|
description | Unique string used to identify the handler. Handler names cannot contain special characters or spaces (validated with Go regex \A[\w\.\-]+\z ). Each handler must have a unique name within its namespace. |
required | true |
type | String |
example |
|
namespace | |
---|---|
description | Sensu RBAC namespace that the handler belongs to. |
required | false |
type | String |
default | default |
example |
|
Spec attributes
command | |
---|---|
description | Handler command to be executed. The event data is passed to the process via stdin.
NOTE: The |
required | true (if type equals pipe ) |
type | String |
example |
|
env_vars | |
---|---|
description | Array of environment variables to use with command execution.
NOTE: The |
required | false |
type | Array |
example |
|
filters | |
---|---|
description | Array of Sensu event filters (by names) to use when filtering events for the handler. Each array item must be a string.
NOTE: We recommend using pipelines, which allow you to list event filters directly in the pipeline resource definition instead of in handlers. |
required | false |
type | Array |
example |
|
handlers | |
---|---|
description | Array of Sensu event handlers (by their names) to use for events using the handler set. Each array item must be a string.
NOTE: The |
required | true (if type equals set ) |
type | Array |
example |
|
mutator | |
---|---|
description | Name of the Sensu event mutator to use to mutate event data for the handler.
NOTE: We recommend using pipelines, which allow you to list mutators directly in the pipeline resource definition instead of in handlers. |
required | false |
type | String |
example |
|
runtime_assets | |
---|---|
description | Array of Sensu dynamic runtime assets (by names) required at runtime to execute the command |
required | false |
type | Array |
example |
|
secrets | |
---|---|
description | Array of the name/secret pairs to use with command execution. |
required | false |
type | Array |
example |
|
socket | |
---|---|
description | Scope for socket definition used to configure the TCP/UDP handler socket.
NOTE: The |
required | true (if type equals tcp or udp ) |
type | Hash |
example |
|
timeout | |
---|---|
description | Handler execution duration timeout (hard stop). In seconds. Only used by pipe , tcp , and udp handler types. |
required | false |
type | Integer |
default | 60 (for tcp and udp handlers) |
example |
|
type | |
---|---|
description | Handler type. |
required | true |
type | String |
allowed values | pipe , tcp , udp , and set |
example |
|
socket
attributes
host | |
---|---|
description | Socket host address (IP or hostname) to connect to. |
required | true |
type | String |
example |
|
port | |
---|---|
description | Socket port to connect to. |
required | true |
type | Integer |
example |
|
secrets
attributes
name | |
---|---|
description | Name of the secret defined in the executable command. Becomes the environment variable presented to the handler. Read Use secrets management in Sensu for more information. |
required | true |
type | String |
example |
|
secret | |
---|---|
description | Name of the Sensu secret resource that defines how to retrieve the secret. |
required | true |
type | String |
example |
|
Send Slack alerts
This handler will send alerts to a channel named monitoring
with the configured webhook URL, using the handler-slack
executable command.
The handler uses the sensu/sensu-slack-handler dynamic runtime asset.
Read Send Slack alerts with handlers for detailed instructions for adding the required asset and configuring this handler.
---
type: Handler
api_version: core/v2
metadata:
name: slack
spec:
command: sensu-slack-handler --channel '#monitoring'
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
handlers: []
runtime_assets:
- sensu/sensu-slack-handler
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "slack"
},
"spec": {
"command": "sensu-slack-handler --channel '#monitoring'",
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
],
"handlers": [],
"runtime_assets": [
"sensu/sensu-slack-handler"
],
"timeout": 0,
"type": "pipe"
}
}
Send registration events
If you configure a Sensu event handler named registration
, the Sensu backend will create and process an event for the agent registration, apply any configured filters and mutators, and execute the registration handler.
Read Automatically register and deregister entities for more information and a registration handler example.
Execute multiple handlers (handler set)
NOTE: We recommend using pipelines to configure multiple workflows for different handlers instead of handler sets.
The following example creates a handler set, notify_all_the_things
, that will execute three handlers: slack
, tcp_handler
, and udp_handler
.
---
type: Handler
api_version: core/v2
metadata:
name: notify_all_the_things
spec:
handlers:
- slack
- tcp_handler
- udp_handler
type: set
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "notify_all_the_things"
},
"spec": {
"type": "set",
"handlers": [
"slack",
"tcp_handler",
"udp_handler"
]
}
}
Use secrets management in a handler
Learn more about secrets management for your Sensu configuration in the secrets and secrets providers references.
---
type: Handler
api_version: core/v2
metadata:
name: ansible-tower
spec:
type: pipe
command: sensu-ansible-handler -h $ANSIBLE_HOST -t $ANSIBLE_TOKEN
secrets:
- name: ANSIBLE_HOST
secret: sensu-ansible-host
- name: ANSIBLE_TOKEN
secret: sensu-ansible-token
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "ansible-tower"
},
"spec": {
"type": "pipe",
"command": "sensu-ansible-handler -h $ANSIBLE_HOST -t $ANSIBLE_TOKEN",
"secrets": [
{
"name": "ANSIBLE_HOST",
"secret": "sensu-ansible-host"
},
{
"name": "ANSIBLE_TOKEN",
"secret": "sensu-ansible-token"
}
]
}
}