Route alerts with event filters
Every alert has an ideal first responder: a team or person who knows how to triage and address the issue. Sensu contact routing lets you alert the right people using their preferred contact methods and reduce mean time to response and recovery.
In this guide, you’ll set up alerts for two teams (dev and ops) with separate Slack channels. Each team wants to be alerted only for the things they care about, using their team’s Slack channel. There’s also a fallback option for alerts that should not be routed to either the dev or ops team. To achieve this, you’ll use a pipeline resource with three workflows, one for each contact option.
Routing alerts requires three types of Sensu resources:
- Handlers to store contact preferences for the dev and ops teams, plus a fallback option
- Event filters to match contact labels to the right handler
- A pipeline to organize the event filters and handlers into workflows that route alerts to the right contacts
Here’s a quick overview of the configuration to set up contact routing with a pipeline.
Two of the check definitions include a contacts
label, which allows the pipeline to route alerts to the correct Slack channel based each workflow’s event filter and handler.
Requirements
To follow this guide, install the Sensu backend, make sure at least one Sensu agent is running, and configure sensuctl to connect to the backend as the admin
user.
You will also need cURL, a Slack webhook URL, and three different Slack channels to receive test alerts (one for each team).
The examples in this guide rely on the check_cpu
check from Monitor server resources with checks.
Before you begin, follow the instructions to add the sensu/check-cpu-usage
dynamic runtime asset and the check_cpu
check.
Configure a Sensu entity
Every Sensu agent has a defined set of subscriptions that determine which checks the agent will execute. For an agent to execute a specific check, you must specify the same subscription in the agent configuration and the check definition.
This guide uses an example check that includes the subscription system
.
Use sensuctl to add a system
subscription to one of your entities.
Before you run the following code, replace <ENTITY_NAME>
with the name of the entity on your system.
NOTE: To find an entity’s name, run sensuctl entity list
.
The ID
is the name of the entity.
sensuctl entity update <ENTITY_NAME>
- For
Entity Class
, press enter. - For
Subscriptions
, typesystem
and press enter.
Run this command to confirm both Sensu services are running:
systemctl status sensu-backend && systemctl status sensu-agent
The response should indicate active (running)
for both the Sensu backend and agent.
Register dynamic runtime assets
Contact routing is powered by the sensu/sensu-go-has-contact-filter dynamic runtime asset. To add the asset to Sensu, use sensuctl asset add:
sensuctl asset add sensu/sensu-go-has-contact-filter:0.3.0 -r contact-filter
The response will indicate that the asset was added:
fetching bonsai asset: sensu/sensu-go-has-contact-filter:0.3.0
added asset: sensu/sensu-go-has-contact-filter:0.3.0
You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["contact-filter"].
This example uses the -r
(rename) flag to specify a shorter name for the asset: contact-filter
.
Next, add the sensu/sensu-slack-handler dynamic runtime asset to Sensu with sensuctl:
sensuctl asset add sensu/sensu-slack-handler:1.5.0 -r sensu-slack-handler
The response will confirm that the asset was added:
fetching bonsai asset: sensu/sensu-slack-handler:1.5.0 -r sensu-slack-handler
added asset: sensu/sensu-slack-handler:1.5.0
You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-slack-handler"].
This example uses the -r
(rename) flag to specify a shorter name for the dynamic runtime asset: sensu-slack-handler
.
Run sensuctl asset list
to confirm that the dynamic runtime assets are ready to use.
The response will confirm the available assets:
Name URL Hash
────────────────────── ───────────────────────────────────────────────────────────────────────────── ──────────
contact-filter //assets.bonsai.sensu.io/.../sensu-go-has-contact-filter_0.3.0.tar.gz d35c6c4
sensu-slack-handler //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_windows_amd64.tar.gz 53359fa
sensu-slack-handler //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_darwin_386.tar.gz e2d7d0d
sensu-slack-handler //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_linux_armv7.tar.gz 362fe51
sensu-slack-handler //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_linux_arm64.tar.gz b492ae2
sensu-slack-handler //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_darwin_amd64.tar.gz 88bbdca
sensu-slack-handler //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_linux_386.tar.gz d9040ae
sensu-slack-handler //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_linux_amd64.tar.gz 6872086
NOTE: Sensu does not download and install dynamic runtime asset builds onto the system until they are needed for command execution.
Create contact filters
The sensu/sensu-go-has-contact-filter dynamic runtime asset supports two functions:
has_contact
, which takes the Sensu event and the contact name as argumentsno_contact
, which is available as a fallback in the absence of contact labels and takes only the event as an argument
You’ll use these functions to create event filters that represent the three actions that the Sensu Slack handler can take on an event: contact the ops team, contact the dev team, and contact the fallback option.
event filter name | expression | description |
---|---|---|
contact_ops |
has_contact(event, "ops") |
Allow events with the entity or check label contacts: ops |
contact_dev |
has_contact(event, "dev") |
Allow events with the entity or check label contacts: dev |
contact_fallback |
no_contacts(event) |
Allow events without an entity or check contacts label |
Use sensuctl to create the three event filters:
echo '---
type: EventFilter
api_version: core/v2
metadata:
name: contact_ops
spec:
action: allow
runtime_assets:
- contact-filter
expressions:
- has_contact(event, "ops")
---
type: EventFilter
api_version: core/v2
metadata:
name: contact_dev
spec:
action: allow
runtime_assets:
- contact-filter
expressions:
- has_contact(event, "dev")
---
type: EventFilter
api_version: core/v2
metadata:
name: contact_fallback
spec:
action: allow
runtime_assets:
- contact-filter
expressions:
- no_contacts(event)' | sensuctl create
echo '{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "contact_ops"
},
"spec": {
"action": "allow",
"runtime_assets": [
"contact-filter"
],
"expressions": [
"has_contact(event, \"ops\")"
]
}
}
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "contact_dev"
},
"spec": {
"action": "allow",
"runtime_assets": [
"contact-filter"
],
"expressions": [
"has_contact(event, \"dev\")"
]
}
}
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "contact_fallback"
},
"spec": {
"action": "allow",
"runtime_assets": [
"contact-filter"
],
"expressions": [
"no_contacts(event)"
]
}
}' | sensuctl create
Use sensuctl to confirm that the event filters were added:
sensuctl filter list
The response should list the new contact_ops
, contact_dev
, and contact_fallback
event filters:
Name Action Expressions
────────────────── ──────── ─────────────────────────────
contact_dev allow (has_contact(event, "dev"))
contact_fallback allow (no_contacts(event))
contact_ops allow (has_contact(event, "ops"))
Create a handler for each contact
With your contact filters in place, you can create a handler for each contact: ops, dev, and fallback. In each handler definition, you will specify:
- A unique name:
ops_handler
,dev_handler
, orfallback_handler
- A customized command with the contact’s preferred Slack channel
- An environment variable that contains your Slack webhook URL
- The
sensu-slack-handler
dynamic runtime asset
Before you run the following code to create the handlers with sensuctl, make these changes:
- Replace
<ALERT_OPS>
,<ALERT_DEV>
, and<ALERT_ALL>
with the names of the channels you want to use to receive alerts in your Slack instance. - Replace
<SLACK_WEBHOOK_URL>
with your Slack webhook URL.
After you update the code to use your preferred Slack channels and webhook URL, run:
echo '---
type: Handler
api_version: core/v2
metadata:
name: ops_handler
spec:
command: sensu-slack-handler --channel "#<ALERT_OPS>"
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx
handlers: null
runtime_assets:
- sensu-slack-handler
secrets: null
timeout: 0
type: pipe
---
type: Handler
api_version: core/v2
metadata:
name: dev_handler
spec:
command: sensu-slack-handler --channel "#<ALERT_DEV>"
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx
handlers: null
runtime_assets:
- sensu-slack-handler
secrets: null
timeout: 0
type: pipe
---
type: Handler
api_version: core/v2
metadata:
name: fallback_handler
spec:
command: sensu-slack-handler --channel "#<ALERT_ALL>"
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx
handlers: null
runtime_assets:
- sensu-slack-handler
secrets: null
timeout: 0
type: pipe' | sensuctl create
echo '{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "ops_handler"
},
"spec": {
"command": "sensu-slack-handler --channel \"#<ALERT_OPS>\"",
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx"
],
"handlers": null,
"runtime_assets": [
"sensu-slack-handler"
],
"secrets": null,
"timeout": 0,
"type": "pipe"
}
}
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "dev_handler"
},
"spec": {
"command": "sensu-slack-handler --channel \"#<ALERT_DEV>\"",
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx"
],
"handlers": null,
"runtime_assets": [
"sensu-slack-handler"
],
"secrets": null,
"timeout": 0,
"type": "pipe"
}
}
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "fallback_handler"
},
"spec": {
"command": "sensu-slack-handler --channel \"#<ALERT_ALL>\"",
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx"
],
"handlers": null,
"runtime_assets": [
"sensu-slack-handler"
],
"secrets": null,
"timeout": 0,
"type": "pipe"
}
}' | sensuctl create
Use sensuctl to confirm that the handlers were added:
sensuctl handler list
The response should list the new dev_handler
, ops_handler
, and fallback_handler
handlers:
Name Type Timeout Filters Mutator Execute Environment Variables Assets
─────────────────── ────── ───────── ───────── ───────── ───────────────────────────────────────────────────────── ───────────────────────────────────────────────────────────── ──────────────────────
dev_handler pipe 0 RUN: sensu-slack-handler --channel "#<ALERT_DEV>" SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx sensu-slack-handler
fallback_handler pipe 0 RUN: sensu-slack-handler --channel "#<ALERT_ALL>" SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx sensu-slack-handler
ops_handler pipe 0 RUN: sensu-slack-handler --channel "#<ALERT_OPS>" SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx sensu-slack-handler
Create a pipeline
Create a pipeline with a three workflows: one for each contact group.
Each workflow includes the contact event filter and the corresponding handler for one contact group. All of the workflows also include the built-in is_incident event filter to reduce noise.
echo '---
type: Pipeline
api_version: core/v2
metadata:
name: slack_contact_routing
spec:
workflows:
- name: dev
filters:
- name: contact_dev
type: EventFilter
api_version: core/v2
- name: is_incident
type: EventFilter
api_version: core/v2
handler:
name: dev_handler
type: Handler
api_version: core/v2
- name: ops
filters:
- name: contact_ops
type: EventFilter
api_version: core/v2
- name: is_incident
type: EventFilter
api_version: core/v2
handler:
name: ops_handler
type: Handler
api_version: core/v2
- name: fallback
filters:
- name: contact_fallback
type: EventFilter
api_version: core/v2
- name: is_incident
type: EventFilter
api_version: core/v2
handler:
name: fallback_handler
type: Handler
api_version: core/v2' | sensuctl create
echo '{
"type": "Pipeline",
"api_version": "core/v2",
"metadata": {
"name": "slack_contact_routing"
},
"spec": {
"workflows": [
{
"name": "dev",
"filters": [
{
"name": "contact_dev",
"type": "EventFilter",
"api_version": "core/v2"
},
{
"name": "is_incident",
"type": "EventFilter",
"api_version": "core/v2"
}
],
"handler": {
"name": "dev_handler",
"type": "Handler",
"api_version": "core/v2"
}
},
{
"name": "ops",
"filters": [
{
"name": "contact_ops",
"type": "EventFilter",
"api_version": "core/v2"
},
{
"name": "is_incident",
"type": "EventFilter",
"api_version": "core/v2"
}
],
"handler": {
"name": "ops_handler",
"type": "Handler",
"api_version": "core/v2"
}
},
{
"name": "fallback",
"filters": [
{
"name": "contact_fallback",
"type": "EventFilter",
"api_version": "core/v2"
},
{
"name": "is_incident",
"type": "EventFilter",
"api_version": "core/v2"
}
],
"handler": {
"name": "fallback_handler",
"type": "Handler",
"api_version": "core/v2"
}
}
]
}
}' | sensuctl create
With your pipeline in place, you can send ad hoc events to test your configuration and make sure the right contact groups receive the right alerts in Slack.
Send events to test your configuration
Use the agent API to create ad hoc events and send them to your Slack pipeline.
First, create an event without a contacts
label.
You may need to modify the URL with your Sensu agent address.
curl -X POST \
-H 'Content-Type: application/json' \
-d '{
"check": {
"metadata": {
"name": "example-check-fallback"
},
"status": 1,
"output": "You should receive this example event in the Slack channel specified by your fallback handler."
},
"pipelines": [
{
"type": "Pipeline",
"api_version": "core/v2",
"name": "contact_routing"
}
]
}' \
http://127.0.0.1:3031/events
Since this event doesn’t include a contacts
label, you should also receive an alert in the Slack channel specified in your fallback_handler
handler.
Behind the scenes, Sensu uses the contact_fallback
filter to match the event to the fallback_handler
handler.
Now, create an event with a contacts
label:
curl -X POST \
-H 'Content-Type: application/json' \
-d '{
"check": {
"metadata": {
"name": "example-check-dev",
"labels": {
"contacts": "dev"
}
},
"status": 1,
"output": "You should receive this example event in the Slack channel specified by your dev handler."
},
"pipelines": [
{
"type": "Pipeline",
"api_version": "core/v2",
"name": "contact_routing"
}
]
}' \
http://127.0.0.1:3031/events
Because this event contains the contacts: dev
label, you should receive an alert in the Slack channel specified by the dev_handler
handler.
Resolve the events by sending the same API requests with status
set to 0
.
Manage contact labels in checks and entities
To assign a check’s alerts to a contact, you can add the contacts
labels to checks or entities.
Route contacts with checks
To test contact routing with check-generated events, update the check_cpu
check to include the ops
and dev
contacts and the slack_contact_routing
pipeline.
Use sensuctl to open the check in a text editor:
sensuctl edit check check_cpu
Edit the check metadata to add the following labels:
labels:
contacts: dev, ops
{
"labels": {
"contacts": "dev, ops"
}
}
Update the pipelines array to add slack_contact_routing
:
pipelines:
- type: Pipeline
api_version: core/v2
name: slack_contact_routing
{
"pipelines": {
"type": "Pipeline",
"api_version": "core/v2",
"name": "slack_contact_routing"
}
}
Save and close the updated check definition. A response will confirm the check was updated. For example:
Updated /api/core/v2/namespaces/default/checks/check_cpu
To view the updated resource definition for check_cpu
and confirm that it includes the contacts
labels and slack_contact_routing
pipeline, run:
sensuctl check info check_cpu --format yaml
sensuctl check info check_cpu --format wrapped-json
The sensuctl response will include the updated check_cpu
resource definition in the specified format:
---
type: CheckConfig
api_version: core/v2
metadata:
created_by: admin
labels:
contacts: dev, ops
name: check_cpu
namespace: default
spec:
check_hooks: null
command: check-cpu-usage -w 75 -c 90
env_vars: null
handlers: []
high_flap_threshold: 0
interval: 60
low_flap_threshold: 0
output_metric_format: ""
output_metric_handlers: null
pipelines:
- api_version: core/v2
name: slack_contact_routing
type: Pipeline
proxy_entity_name: ""
publish: true
round_robin: false
runtime_assets:
- check-cpu-usage
secrets: null
stdin: false
subdue: null
subscriptions:
- system
timeout: 0
ttl: 0
{
"type": "CheckConfig",
"api_version": "core/v2",
"metadata": {
"name": "check_cpu",
"namespace": "default",
"labels": {
"contacts": "dev, ops"
},
"created_by": "admin"
},
"spec": {
"check_hooks": null,
"command": "check-cpu-usage -w 75 -c 90",
"env_vars": null,
"handlers": [],
"high_flap_threshold": 0,
"interval": 60,
"low_flap_threshold": 0,
"output_metric_format": "",
"output_metric_handlers": null,
"pipelines": [
{
"api_version": "core/v2",
"name": "slack_contact_routing",
"type": "Pipeline"
}
],
"proxy_entity_name": "",
"publish": true,
"round_robin": false,
"runtime_assets": [
"check-cpu-usage"
],
"secrets": null,
"stdin": false,
"subdue": null,
"subscriptions": [
"system"
],
"timeout": 0,
"ttl": 0
}
}
Now when the check_cpu
check generates an event, Sensu will filter the event according to the contact_dev
and contact_ops
event filters and send alerts to the #dev and #ops Slack channels:
Entities
You can specify contacts in entity labels instead of in check labels. The check definition should still include the pipeline.
If contact labels are present in both the check and entity, the check contacts override the entity contacts.
In this example, the dev
label in the check configuration overrides the ops
label in the agent definition, resulting in an alert sent to #dev but not to #ops or #fallback:
What’s next
Now that you’ve set up contact routing for two example teams, you can create additional filters, handlers, and labels to represent your team’s contacts. Learn how to use Sensu to Reduce alert fatigue.
Read more about the Sensu features you used in this guide:
- Subscriptions
- sensuctl
- Dynamic runtime assets, Bonsai, sensu/sensu-go-has-contact-filter, and sensu/sensu-slack-handler
- is_incident event filter
- Agent API
- Entity labels
Save the event filter, handler, and check definitions you created in this guide to YAML or JSON files to start developing a monitoring as code repository. Storing your Sensu configurations the same way you would store code means they are portable and repeatable. Monitoring as code makes it possible to reproduce an environment’s configuration and move to a more robust deployment without losing what you’ve started.