Handlers reference

Sensu executes handlers during the process stage of the observability pipeline.

Handlers are actions the Sensu backend executes on events. Several types of handlers are available. The most common are pipe handlers, which work similarly to checks and enable Sensu to interact with almost any computer program via standard streams.

Pipe handlers send observation data (events) into arbitrary commands via stdin
TCP/UDP handlers send observation data (events) to a remote socket
Handler sets group event handlers and streamline groups of actions to execute for certain types of events (also called “set handlers”)

The handler stack concept describes a group of handlers or a handler set that escalates events through a series of different handlers.

Discover, download, and share Sensu handler dynamic runtime assets using Bonsai, the Sensu asset hub. Read Use dynamic runtime assets to install plugins to get started.

Pipe handlers

Pipe handlers are external commands that can consume event data via stdin.

Pipe handler example

This example shows a pipe handler resource definition with the minimum required attributes:

---
type: Handler
api_version: core/v2
metadata:
  name: pipe_handler_minimum
spec:
  command: command-example
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "pipe_handler_minimum"
  },
  "spec": {
    "command": "command-example",
    "type": "pipe"
  }
}

Pipe handler command

Pipe handler definitions include a command attribute, which is a command for the Sensu backend to execute.

Pipe handler command arguments

Pipe handler command attributes may include command line arguments for controlling the behavior of the command executable.

TCP/UDP handlers

TCP and UDP handlers enable Sensu to forward event data to arbitrary TCP or UDP sockets for external services to consume.

TCP/UDP handler example

This handler will send event data to a TCP socket (10.0.1.99:4444) and timeout if an acknowledgement (ACK) is not received within 30 seconds:

---
type: Handler
api_version: core/v2
metadata:
  name: tcp_handler
spec:
  socket:
    host: 10.0.1.99
    port: 4444
  type: tcp
  timeout: 30

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "tcp_handler"
  },
  "spec": {
    "type": "tcp",
    "timeout": 30,
    "socket": {
      "host": "10.0.1.99",
      "port": 4444
    }
  }
}

Change the type from tcp to udp to configure a UDP handler:

---
type: Handler
api_version: core/v2
metadata:
  name: udp_handler
spec:
  socket:
    host: 10.0.1.99
    port: 4444
  type: udp
  timeout: 30

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "udp_handler"
  },
  "spec": {
    "type": "udp",
    "timeout": 30,
    "socket": {
      "host": "10.0.1.99",
      "port": 4444
    }
  }
}

Handler sets

NOTE: We recommend using pipelines to configure multiple workflows for different handlers instead of handler sets.

Handler set definitions allow you to use a single named handler set to refer to groups of handlers. The handler set becomes a collection of individual actions to take (via each included handler) on event data.

For example, suppose you have already created these two handlers:

elasticsearch to send all observation data to Elasticsearch.
opsgenie to send non-OK status alerts to your OpsGenie notification channel.

You can list both of these handlers in a handler set to automate and streamline your workflow, specifying type: set:

---
type: Handler
api_version: core/v2
metadata:
  name: send_events_notify_operator
spec:
  handlers:
  - elasticsearch
  - opsgenie
  type: set

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "send_events_notify_operator"
  },
  "spec": {
    "type": "set",
    "handlers": [
      "elasticsearch",
      "opsgenie"
    ]
  }
}

Now you can route observation data to Elasticsearch and alerts to OpsGenie with a single handler definition, the send_events_notify_operator handler set.

NOTE: Attributes defined in handler sets do not apply to the handlers they include. For example, filters and mutator attributes defined in a handler set will have no effect on handlers. Define these attributes in individual handlers instead, or use pipelines.

Handler stacks

NOTE: We recommend using pipelines to configure multiple workflows for escalating events through a series of handlers instead of handler stacks.

The handler stack concept refers to a group of handlers or a handler set that escalates events through a series of different handlers. For example, suppose you want a handler stack with three levels of escalation:

Level 1: On the first occurrence, attempt remediation.
Level 2: On the fifth occurrence, send an alert to Slack.
Level 3: On the tenth occurrence, send an alert to PagerDuty. Continue to send this alert on every tenth occurrence thereafter until the incident is resolved.

A handler stack for this scenario requires three handlers to take the desired actions based on three corresponding event filters that control the escalation levels:

Level 1 requires an event filter with the built-in is_incident filter plus an occurrence-based filter that uses an expression like event.check.occurrences ==1 and a corresponding remediation handler.
Level 2 requires an event filter with is_incident plus an occurrence-based filter that uses an expression like event.check.occurrences == 5 and a corresponding Slack handler.
Level 3 requires an event filter with is_incident plus an occurrence-based filter that uses an expression like event.check.occurrences % 10 == 0 to match event data with an occurrences value that is evenly divisible by 10 via a modulo operator calculation and a corresponding PagerDuty handler.

With these event filters and handlers configured, you can create a handler set that includes the three handlers in your stack. You can also list the three handlers in the handlers array in your check definition instead.

PRO TIP: This scenario relies on six different resources, three event filters and three handlers, to describe the handler stack concept, but you can use Sensu dynamic runtime assets and integrations to achieve the same escalating alert levels in other ways.

For example, you can use the is_incident event filter in conjunction with the sensu/sensu-go-fatigue-check-filter asset to control event escalation. The sensu/sensu-ansible-handler, sensu/sensu-rundeck-handler, and sensu/sensu-saltstack-handler auto-remediation integrations and the sensu/sensu-remediation-handler asset also include built-in occurrence- and severity-based event filtering.

Keepalive event handlers

Sensu keepalives are the heartbeat mechanism used to ensure that all registered Sensu agents are operational and can reach the Sensu backend. You can connect keepalive events to your monitoring workflows using a keepalive handler. Sensu looks for an event handler named keepalive and automatically uses it to process keepalive events.

Suppose you want to receive Slack notifications for keepalive alerts, and you already have a Slack handler set up to process events. To process keepalive events using the Slack pipeline, create a handler set named keepalive and add the slack handler to the handlers array. The resulting keepalive handler set configuration will look like this example:

---
type: Handler
api_version: core/v2
metadata:
  name: keepalive
spec:
  handlers:
  - slack
  type: set

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "keepalive"
  },
  "spec": {
    "type": "set",
    "handlers": [
      "slack"
    ]
  }
}

You can also use the keepalive-handlers configuration option to send keepalive events to any handler you have configured. If you do not specify a keepalive handler with the keepalive-handlers configuration option, the Sensu backend will use the default keepalive handler and create an event in sensuctl and the Sensu web UI.

Handler specification

Top-level attributes

api_version
description	Top-level attribute that specifies the Sensu API group and version. For handlers in this version of Sensu, the `api_version` should always be `core/v2`.
required	Required for handler definitions in `wrapped-json` or `yaml` format for use with `sensuctl create`.
type	String
example	`api_version: core/v2` `{ "api_version": "core/v2" }`

metadata
description	Top-level collection of metadata about the handler that includes `name`, `namespace`, and `created_by` as well as custom `labels` and `annotations`. The `metadata` map is always at the top level of the handler definition. This means that in `wrapped-json` and `yaml` formats, the `metadata` scope occurs outside the `spec` scope. Read metadata attributes for details.
required	Required for handler definitions in `wrapped-json` or `yaml` format for use with `sensuctl create`.
type	Map of key-value pairs
example	`metadata: name: handler-slack namespace: default created_by: admin labels: region: us-west-1 annotations: slack-channel: "#monitoring"` `{ "metadata": { "name": "handler-slack", "namespace": "default", "created_by": "admin", "labels": { "region": "us-west-1" }, "annotations": { "slack-channel": "#monitoring" } } }`

spec
description	Top-level map that includes the handler spec attributes.
required	Required for handler definitions in `wrapped-json` or `yaml` format for use with `sensuctl create`.
type	Map of key-value pairs
example	`spec: type: tcp socket: host: 10.0.1.99 port: 4444 metadata: name: tcp_handler namespace: default` `{ "spec": { "type": "tcp", "socket": { "host": "10.0.1.99", "port": 4444 }, "metadata": { "name": "tcp_handler", "namespace": "default" } } }`

type
description	Top-level attribute that specifies the `sensuctl create` resource type. Handlers should always be type `Handler`.
required	Required for handler definitions in `wrapped-json` or `yaml` format for use with `sensuctl create`.
type	String
example	`type: Handler` `{ "type": "Handler" }`

Metadata attributes

annotations
description	Non-identifying metadata to include with observation event data that you can access with event filters. You can use annotations to add data that’s meaningful to people or external tools that interact with Sensu. In contrast to labels, you cannot use annotations in API response filtering, sensuctl response filtering, or web UI views.
required	false
type	Map of key-value pairs. Keys and values can be any valid UTF-8 string.
default	`null`
example	`annotations: managed-by: ops playbook: www.example.url` `{ "annotations": { "managed-by": "ops", "playbook": "www.example.url" } }`

created_by
description	Username of the Sensu user who created the handler or last updated the handler. Sensu automatically populates the `created_by` field when the handler is created or updated.
required	false
type	String
example	`created_by: admin` `{ "created_by": "admin" }`

labels
description	Custom attributes to include with observation event data that you can use for response and web UI view filtering. If you include labels in your event data, you can filter API responses, sensuctl responses, and web UI views based on them. In other words, labels allow you to create meaningful groupings for your data. Limit labels to metadata you need to use for response filtering. For complex, non-identifying metadata that you will not need to use in response filtering, use annotations rather than labels.
required	false
type	Map of key-value pairs. Keys can contain only letters, numbers, and underscores and must start with a letter. Values can be any valid UTF-8 string.
default	`null`
example	`labels: environment: development region: us-west-2` `{ "labels": { "environment": "development", "region": "us-west-2" } }`

name
description	Unique string used to identify the handler. Handler names cannot contain special characters or spaces (validated with Go regex `\A[\w\.\-]+\z`). Each handler must have a unique name within its namespace.
required	true
type	String
example	`name: handler-slack` `{ "name": "handler-slack" }`

namespace
description	Sensu RBAC namespace that the handler belongs to.
required	false
type	String
default	`default`
example	`namespace: production` `{ "namespace": "production" }`

Spec attributes

command
description	Handler command to be executed. The event data is passed to the process via stdin. NOTE: The `command` attribute is only supported for pipe handlers (that is, handlers configured with `"type": "pipe"`).
required	true (if `type` equals `pipe`)
type	String
example	`command: /etc/sensu/plugins/pagerduty.go` `{ "command": "/etc/sensu/plugins/pagerduty.go" }`

env_vars
description	Array of environment variables to use with command execution. NOTE: The `env_vars` attribute is only supported for pipe handlers (that is, handlers configured with `"type": "pipe"`).
required	false
type	Array
example	`env_vars: - API_KEY=0428d6b8nb51an4d95nbe28nf90865a66af5` `{ "env_vars": [ "API_KEY=0428d6b8nb51an4d95nbe28nf90865a66af5" ] }`

filters
description	Array of Sensu event filters (by names) to use when filtering events for the handler. Each array item must be a string. NOTE: We recommend using pipelines, which allow you to list event filters directly in the pipeline resource definition instead of in handlers. Pipelines ignore any event filters specified in handler definitions, so you do not need to remove them to use your existing handlers — just make sure to define the event filters you want to use in the pipeline workflow.
required	false
type	Array
example	`filters: - is_incident - not_silenced - state_change_only` `{ "filters": [ "is_incident", "not_silenced", "state_change_only" ] }`

handlers
description	Array of Sensu event handlers (by their names) to use for events using the handler set. Each array item must be a string. NOTE: The `handlers` attribute is only supported for handler sets (that is, handlers configured with `"type": "set"`). We recommend using pipelines to configure multiple workflows instead of handler sets.
required	true (if `type` equals `set`)
type	Array
example	`handlers: - pagerduty - email - ec2` `{ "handlers": [ "pagerduty", "email", "ec2" ] }`

mutator
description	Name of the Sensu event mutator to use to mutate event data for the handler. NOTE: We recommend using pipelines, which allow you to list mutators directly in the pipeline resource definition instead of in handlers. Pipelines ignore any mutators specified in handler definitions, so you do not need to remove them to use your existing handlers — just make sure to define the mutator you want to use in the pipeline workflow.
required	false
type	String
example	`mutator: only_check_output` `{ "mutator": "only_check_output" }`

runtime_assets
description	Array of Sensu dynamic runtime assets (by names) required at runtime to execute the `command`
required	false
type	Array
example	`runtime_assets: - metric-handler` `{ "runtime_assets": [ "metric-handler" ] }`

secrets
description	Array of the name/secret pairs to use with command execution.
required	false
type	Array
example	`secrets: - name: ANSIBLE_HOST secret: sensu-ansible-host - name: ANSIBLE_TOKEN secret: sensu-ansible-token` `{ "secrets": [ { "name": "ANSIBLE_HOST", "secret": "sensu-ansible-host" }, { "name": "ANSIBLE_TOKEN", "secret": "sensu-ansible-token" } ] }`

socket
description	Scope for `socket` definition used to configure the TCP/UDP handler socket. NOTE: The `socket` attribute is only supported for TCP/UDP handlers (that is, handlers configured with `"type": "tcp"` or `"type": "udp"`).
required	true (if `type` equals `tcp` or `udp`)
type	Hash
example	`socket: {}` `{ "socket": {} }`

timeout
description	Handler execution duration timeout (hard stop). In seconds. Only used by `pipe`, `tcp`, and `udp` handler types.
required	false
type	Integer
default	`60` (for `tcp` and `udp` handlers)
example	`timeout: 30` `{ "timeout": 30 }`

type
description	Handler type.
required	true
type	String
allowed values	`pipe`, `tcp`, `udp`, and `set`
example	`type: pipe` `{ "type": "pipe" }`

`socket` attributes

host
description	Socket host address (IP or hostname) to connect to.
required	true
type	String
example	`host: 8.8.8.8` `{ "host": "8.8.8.8" }`

port
description	Socket port to connect to.
required	true
type	Integer
example	`port: 4242` `{ "port": 4242 }`

`secrets` attributes

name
description	Name of the secret defined in the executable command. Becomes the environment variable presented to the handler. Read Use secrets management in Sensu for more information.
required	true
type	String
example	`name: ANSIBLE_HOST` `{ "name": "ANSIBLE_HOST" }`

secret
description	Name of the Sensu secret resource that defines how to retrieve the secret.
required	true
type	String
example	`secret: sensu-ansible-host` `{ "secret": "sensu-ansible-host" }`

Send Slack alerts

This handler will send alerts to a channel named monitoring with the configured webhook URL, using the handler-slack executable command. The handler uses the sensu/sensu-slack-handler dynamic runtime asset. Read Send Slack alerts with handlers for detailed instructions for adding the required asset and configuring this handler.

---
type: Handler
api_version: core/v2
metadata:
  name: slack
spec:
  command: sensu-slack-handler --channel '#monitoring'
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
  handlers: []
  runtime_assets:
  - sensu/sensu-slack-handler
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack"
  },
  "spec": {
    "command": "sensu-slack-handler --channel '#monitoring'",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
    ],
    "handlers": [],
    "runtime_assets": [
      "sensu/sensu-slack-handler"
    ],
    "timeout": 0,
    "type": "pipe"
  }
}

Send registration events

If you configure a Sensu event handler named registration, the Sensu backend will create and process an event for the agent registration, apply any configured filters and mutators, and execute the registration handler.

Read Automatically register and deregister entities for more information and a registration handler example.

Execute multiple handlers (handler set)

NOTE: We recommend using pipelines to configure multiple workflows for different handlers instead of handler sets.

The following example creates a handler set, notify_all_the_things, that will execute three handlers: slack, tcp_handler, and udp_handler.

---
type: Handler
api_version: core/v2
metadata:
  name: notify_all_the_things
spec:
  handlers:
  - slack
  - tcp_handler
  - udp_handler
  type: set

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "notify_all_the_things"
  },
  "spec": {
    "type": "set",
    "handlers": [
      "slack",
      "tcp_handler",
      "udp_handler"
    ]
  }
}

Use secrets management in a handler

Learn more about secrets management for your Sensu configuration in the secrets and secrets providers references.

---
type: Handler 
api_version: core/v2 
metadata:
  name: ansible-tower
spec: 
  type: pipe
  command: sensu-ansible-handler -h $ANSIBLE_HOST -t $ANSIBLE_TOKEN
  secrets:
  - name: ANSIBLE_HOST
    secret: sensu-ansible-host
  - name: ANSIBLE_TOKEN
    secret: sensu-ansible-token

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "ansible-tower"
  },
  "spec": {
    "type": "pipe",
    "command": "sensu-ansible-handler -h $ANSIBLE_HOST -t $ANSIBLE_TOKEN",
    "secrets": [
      {
        "name": "ANSIBLE_HOST",
        "secret": "sensu-ansible-host"
      },
      {
        "name": "ANSIBLE_TOKEN",
        "secret": "sensu-ansible-token"
      }
    ]
  }
}

Handlers reference

Pipe handlers

Pipe handler example

Pipe handler command

Pipe handler command arguments

TCP/UDP handlers

TCP/UDP handler example

Handler sets

Handler stacks

Keepalive event handlers

Handler specification

Top-level attributes

Metadata attributes

Spec attributes

`socket` attributes

`secrets` attributes

Send Slack alerts

Send registration events

Execute multiple handlers (handler set)

Use secrets management in a handler

Register your Sensu installation — Get rewards!

About Sensu

Navigation

Resources

Community

Get In Touch

Join Our Community

Handlers reference

Pipe handlers

Pipe handler example

Pipe handler command

Pipe handler command arguments

TCP/UDP handlers

TCP/UDP handler example

Handler sets

Handler stacks

Keepalive event handlers

Handler specification

Top-level attributes

Metadata attributes

Spec attributes

socket attributes

secrets attributes

Send Slack alerts

Send registration events

Execute multiple handlers (handler set)

Use secrets management in a handler

Register your Sensu installation — Get rewards!

About Sensu

Navigation

Resources

Community

Get In Touch

Join Our Community

`socket` attributes

`secrets` attributes