Route alerts with event filters

Every alert has an ideal first responder: a team or person who knows how to triage and address the issue. Sensu contact routing lets you alert the right people using their preferred contact methods and reduce mean time to response and recovery.

In this guide, you’ll set up alerts for two teams (dev and ops) with separate Slack channels. Each team wants to be alerted only for the things they care about, using their team’s Slack channel. There’s also a fallback option for alerts that should not be routed to either the dev or ops team. To achieve this, you’ll use a pipeline resource with three workflows, one for each contact option.

Routing alerts requires three types of Sensu resources:

  • Handlers to store contact preferences for the dev and ops teams, plus a fallback option
  • Event filters to match contact labels to the right handler
  • A pipeline to organize the event filters and handlers into workflows that route alerts to the right contacts

Here’s a quick overview of the configuration to set up contact routing with a pipeline. Two of the check definitions include a contacts label, which allows the pipeline to route alerts to the correct Slack channel based each workflow’s event filter and handler.

Diagram that shows events generated with and without labels, matched to the appropriate handler using a contact filter and routed to the appropriate Slack channel

Requirements

To follow this guide, install the Sensu backend, make sure at least one Sensu agent is running, and configure sensuctl to connect to the backend as the admin user.

You will also need cURL, a Slack webhook URL, and three different Slack channels to receive test alerts (one for each team).

The examples in this guide rely on the check_cpu check from Monitor server resources with checks. Before you begin, follow the instructions to add the sensu/check-cpu-usage dynamic runtime asset and the check_cpu check.

Configure a Sensu entity

Every Sensu agent has a defined set of subscriptions that determine which checks the agent will execute. For an agent to execute a specific check, you must specify the same subscription in the agent configuration and the check definition.

This guide uses an example check that includes the subscription system. Use sensuctl to add a system subscription to one of your entities.

Before you run the following code, replace <ENTITY_NAME> with the name of the entity on your system.

NOTE: To find an entity’s name, run sensuctl entity list. The ID is the name of the entity.

sensuctl entity update <ENTITY_NAME>
  • For Entity Class, press enter.
  • For Subscriptions, type system and press enter.

Run this command to confirm both Sensu services are running:

systemctl status sensu-backend && systemctl status sensu-agent

The response should indicate active (running) for both the Sensu backend and agent.

Register dynamic runtime assets

Contact routing is powered by the sensu/sensu-go-has-contact-filter dynamic runtime asset. To add the asset to Sensu, use sensuctl asset add:

sensuctl asset add sensu/sensu-go-has-contact-filter:0.3.0 -r contact-filter

The response will indicate that the asset was added:

fetching bonsai asset: sensu/sensu-go-has-contact-filter:0.3.0
added asset: sensu/sensu-go-has-contact-filter:0.3.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["contact-filter"].

This example uses the -r (rename) flag to specify a shorter name for the asset: contact-filter.

Next, add the sensu/sensu-slack-handler dynamic runtime asset to Sensu with sensuctl:

sensuctl asset add sensu/sensu-slack-handler:1.5.0 -r sensu-slack-handler

The response will confirm that the asset was added:

fetching bonsai asset: sensu/sensu-slack-handler:1.5.0 -r sensu-slack-handler
added asset: sensu/sensu-slack-handler:1.5.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-slack-handler"].

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: sensu-slack-handler.

Run sensuctl asset list to confirm that the dynamic runtime assets are ready to use. The response will confirm the available assets:

         Name                                               URL                                        Hash    
────────────────────── ───────────────────────────────────────────────────────────────────────────── ──────────
  contact-filter        //assets.bonsai.sensu.io/.../sensu-go-has-contact-filter_0.3.0.tar.gz         d35c6c4  
  sensu-slack-handler   //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_windows_amd64.tar.gz   53359fa  
  sensu-slack-handler   //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_darwin_386.tar.gz      e2d7d0d  
  sensu-slack-handler   //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_linux_armv7.tar.gz     362fe51  
  sensu-slack-handler   //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_linux_arm64.tar.gz     b492ae2  
  sensu-slack-handler   //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_darwin_amd64.tar.gz    88bbdca  
  sensu-slack-handler   //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_linux_386.tar.gz       d9040ae  
  sensu-slack-handler   //assets.bonsai.sensu.io/.../sensu-slack-handler_1.0.3_linux_amd64.tar.gz     6872086

NOTE: Sensu does not download and install dynamic runtime asset builds onto the system until they are needed for command execution.

Create contact filters

The sensu/sensu-go-has-contact-filter dynamic runtime asset supports two functions:

  • has_contact, which takes the Sensu event and the contact name as arguments
  • no_contact, which is available as a fallback in the absence of contact labels and takes only the event as an argument

You’ll use these functions to create event filters that represent the three actions that the Sensu Slack handler can take on an event: contact the ops team, contact the dev team, and contact the fallback option.

event filter name expression description
contact_ops has_contact(event, "ops") Allow events with the entity or check label contacts: ops
contact_dev has_contact(event, "dev") Allow events with the entity or check label contacts: dev
contact_fallback no_contacts(event) Allow events without an entity or check contacts label

Use sensuctl to create the three event filters:

echo '---
type: EventFilter
api_version: core/v2
metadata:
  name: contact_ops
spec:
  action: allow
  runtime_assets:
    - contact-filter
  expressions:
    - has_contact(event, "ops")
---
type: EventFilter
api_version: core/v2
metadata:
  name: contact_dev
spec:
  action: allow
  runtime_assets:
    - contact-filter
  expressions:
    - has_contact(event, "dev")
---
type: EventFilter
api_version: core/v2
metadata:
  name: contact_fallback
spec:
  action: allow
  runtime_assets:
    - contact-filter
  expressions:
    - no_contacts(event)' | sensuctl create
echo '{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "contact_ops"
  },
  "spec": {
    "action": "allow",
    "runtime_assets": [
      "contact-filter"
    ],
    "expressions": [
      "has_contact(event, \"ops\")"
    ]
  }
}
{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "contact_dev"
  },
  "spec": {
    "action": "allow",
    "runtime_assets": [
      "contact-filter"
    ],
    "expressions": [
      "has_contact(event, \"dev\")"
    ]
  }
}
{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "contact_fallback"
  },
  "spec": {
    "action": "allow",
    "runtime_assets": [
      "contact-filter"
    ],
    "expressions": [
      "no_contacts(event)"
    ]
  }
}' | sensuctl create

Use sensuctl to confirm that the event filters were added:

sensuctl filter list

The response should list the new contact_ops, contact_dev, and contact_fallback event filters:

        Name         Action           Expressions          
 ────────────────── ──────── ───────────────────────────── 
  contact_dev        allow    (has_contact(event, "dev"))  
  contact_fallback   allow    (no_contacts(event))         
  contact_ops        allow    (has_contact(event, "ops"))

Create a handler for each contact

With your contact filters in place, you can create a handler for each contact: ops, dev, and fallback. In each handler definition, you will specify:

  • A unique name: ops_handler, dev_handler, or fallback_handler
  • A customized command with the contact’s preferred Slack channel
  • An environment variable that contains your Slack webhook URL
  • The sensu-slack-handler dynamic runtime asset

Before you run the following code to create the handlers with sensuctl, make these changes:

  • Replace <ALERT_OPS>, <ALERT_DEV>, and <ALERT_ALL> with the names of the channels you want to use to receive alerts in your Slack instance.
  • Replace <SLACK_WEBHOOK_URL> with your Slack webhook URL.

After you update the code to use your preferred Slack channels and webhook URL, run:

echo '---
type: Handler
api_version: core/v2
metadata:
  name: ops_handler
spec:
  command: sensu-slack-handler --channel "#<ALERT_OPS>"
  env_vars:
    - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx
  handlers: null
  runtime_assets:
    - sensu-slack-handler
  secrets: null
  timeout: 0
  type: pipe
---
type: Handler
api_version: core/v2
metadata:
  name: dev_handler
spec:
  command: sensu-slack-handler --channel "#<ALERT_DEV>"
  env_vars:
    - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx
  handlers: null
  runtime_assets:
    - sensu-slack-handler
  secrets: null
  timeout: 0
  type: pipe
---
type: Handler
api_version: core/v2
metadata:
  name: fallback_handler
spec:
  command: sensu-slack-handler --channel "#<ALERT_ALL>"
  env_vars:
    - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx
  handlers: null
  runtime_assets:
    - sensu-slack-handler
  secrets: null
  timeout: 0
  type: pipe' | sensuctl create
echo '{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "ops_handler"
  },
  "spec": {
    "command": "sensu-slack-handler --channel \"#<ALERT_OPS>\"",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx"
    ],
    "handlers": null,
    "runtime_assets": [
      "sensu-slack-handler"
    ],
    "secrets": null,
    "timeout": 0,
    "type": "pipe"
  }
}
{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "dev_handler"
  },
  "spec": {
    "command": "sensu-slack-handler --channel \"#<ALERT_DEV>\"",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx"
    ],
    "handlers": null,
    "runtime_assets": [
      "sensu-slack-handler"
    ],
    "secrets": null,
    "timeout": 0,
    "type": "pipe"
  }
}
{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "fallback_handler"
  },
  "spec": {
    "command": "sensu-slack-handler --channel \"#<ALERT_ALL>\"",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx"
    ],
    "handlers": null,
    "runtime_assets": [
      "sensu-slack-handler"
    ],
    "secrets": null,
    "timeout": 0,
    "type": "pipe"
  }
}' | sensuctl create

Use sensuctl to confirm that the handlers were added:

sensuctl handler list

The response should list the new dev_handler, ops_handler, and fallback_handler handlers:

        Name         Type   Timeout   Filters   Mutator                       Execute                                                   Environment Variables                            Assets         
─────────────────── ────── ───────── ───────── ───────── ───────────────────────────────────────────────────────── ───────────────────────────────────────────────────────────── ──────────────────────
  dev_handler        pipe         0                       RUN:  sensu-slack-handler --channel "#<ALERT_DEV>"        SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx   sensu-slack-handler  
  fallback_handler   pipe         0                       RUN:  sensu-slack-handler --channel "#<ALERT_ALL>"   SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx   sensu-slack-handler  
  ops_handler        pipe         0                       RUN:  sensu-slack-handler --channel "#<ALERT_OPS>"        SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxxxxxxx   sensu-slack-handler

Create a pipeline

Create a pipeline with a three workflows: one for each contact group.

Each workflow includes the contact event filter and the corresponding handler for one contact group. All of the workflows also include the built-in is_incident event filter to reduce noise.

echo '---
type: Pipeline
api_version: core/v2
metadata:
  name: slack_contact_routing
spec:
  workflows:
  - name: dev
    filters:
    - name: contact_dev
      type: EventFilter
      api_version: core/v2
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    handler:
      name: dev_handler
      type: Handler
      api_version: core/v2
  - name: ops
    filters:
    - name: contact_ops
      type: EventFilter
      api_version: core/v2
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    handler:
      name: ops_handler
      type: Handler
      api_version: core/v2
  - name: fallback
    filters:
    - name: contact_fallback
      type: EventFilter
      api_version: core/v2
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    handler:
      name: fallback_handler
      type: Handler
      api_version: core/v2' | sensuctl create
echo '{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack_contact_routing"
  },
  "spec": {
    "workflows": [
      {
        "name": "dev",
        "filters": [
          {
            "name": "contact_dev",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "dev_handler",
          "type": "Handler",
          "api_version": "core/v2"
        }
      },
      {
        "name": "ops",
        "filters": [
          {
            "name": "contact_ops",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "ops_handler",
          "type": "Handler",
          "api_version": "core/v2"
        }
      },
      {
        "name": "fallback",
        "filters": [
          {
            "name": "contact_fallback",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "fallback_handler",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}' | sensuctl create

With your pipeline in place, you can send ad hoc events to test your configuration and make sure the right contact groups receive the right alerts in Slack.

Send events to test your configuration

Use the agent API to create ad hoc events and send them to your Slack pipeline.

First, create an event without a contacts label. You may need to modify the URL with your Sensu agent address.

curl -X POST \
-H 'Content-Type: application/json' \
-d '{
  "check": {
    "metadata": {
      "name": "example-check-fallback"
    },
    "status": 1,
    "output": "You should receive this example event in the Slack channel specified by your fallback handler."
  },
  "pipelines": [
    {
      "type": "Pipeline",
      "api_version": "core/v2",
      "name": "contact_routing"
    }
  ]
}' \
http://127.0.0.1:3031/events

Since this event doesn’t include a contacts label, you should also receive an alert in the Slack channel specified in your fallback_handler handler. Behind the scenes, Sensu uses the contact_fallback filter to match the event to the fallback_handler handler.

Now, create an event with a contacts label:

curl -X POST \
-H 'Content-Type: application/json' \
-d '{
  "check": {
    "metadata": {
      "name": "example-check-dev",
      "labels": {
        "contacts": "dev"
      }
    },
    "status": 1,
    "output": "You should receive this example event in the Slack channel specified by your dev handler."
  },
  "pipelines": [
    {
      "type": "Pipeline",
      "api_version": "core/v2",
      "name": "contact_routing"
    }
  ]
}' \
http://127.0.0.1:3031/events

Because this event contains the contacts: dev label, you should receive an alert in the Slack channel specified by the dev_handler handler.

Resolve the events by sending the same API requests with status set to 0.

Manage contact labels in checks and entities

To assign a check’s alerts to a contact, you can add the contacts labels to checks or entities.

Route contacts with checks

To test contact routing with check-generated events, update the check_cpu check to include the ops and dev contacts and the slack_contact_routing pipeline.

Use sensuctl to open the check in a text editor:

sensuctl edit check check_cpu

Edit the check metadata to add the following labels:

labels:
  contacts: dev, ops
{
  "labels": {
    "contacts": "dev, ops"
  }
}

Update the pipelines array to add slack_contact_routing:

pipelines:
  - type: Pipeline
    api_version: core/v2
    name: slack_contact_routing
{
  "pipelines": {
    "type": "Pipeline",
    "api_version": "core/v2",
    "name": "slack_contact_routing"
  }
}

Save and close the updated check definition. A response will confirm the check was updated. For example:

Updated /api/core/v2/namespaces/default/checks/check_cpu

To view the updated resource definition for check_cpu and confirm that it includes the contacts labels and slack_contact_routing pipeline, run:

sensuctl check info check_cpu --format yaml
sensuctl check info check_cpu --format wrapped-json

The sensuctl response will include the updated check_cpu resource definition in the specified format:

---
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  labels:
    contacts: dev, ops
  name: check_cpu
  namespace: default
spec:
  check_hooks: null
  command: check-cpu-usage -w 75 -c 90
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  pipelines:
  - api_version: core/v2
    name: slack_contact_routing
    type: Pipeline
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - check-cpu-usage
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system
  timeout: 0
  ttl: 0
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "check_cpu",
    "namespace": "default",
    "labels": {
      "contacts": "dev, ops"
    },
    "created_by": "admin"
  },
  "spec": {
    "check_hooks": null,
    "command": "check-cpu-usage -w 75 -c 90",
    "env_vars": null,
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 60,
    "low_flap_threshold": 0,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "pipelines": [
      {
        "api_version": "core/v2",
        "name": "slack_contact_routing",
        "type": "Pipeline"
      }
    ],
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "check-cpu-usage"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "system"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

Now when the check_cpu check generates an event, Sensu will filter the event according to the contact_dev and contact_ops event filters and send alerts to the #dev and #ops Slack channels:

Diagram that shows an event generated with a check label for the dev and ops teams, matched to the dev team and ops team handlers using contact filters, and routed to the Slack channels for dev and ops

Entities

You can specify contacts in entity labels instead of in check labels. The check definition should still include the pipeline.

If contact labels are present in both the check and entity, the check contacts override the entity contacts. In this example, the dev label in the check configuration overrides the ops label in the agent definition, resulting in an alert sent to #dev but not to #ops or #fallback:

Diagram that shows how check labels override entity labels when both are present in an event

What’s next

Now that you’ve set up contact routing for two example teams, you can create additional filters, handlers, and labels to represent your team’s contacts. Learn how to use Sensu to Reduce alert fatigue.

Read more about the Sensu features you used in this guide:

Save the event filter, handler, and check definitions you created in this guide to YAML or JSON files to start developing a monitoring as code repository. Storing your Sensu configurations the same way you would store code means they are portable and repeatable. Monitoring as code makes it possible to reproduce an environment’s configuration and move to a more robust deployment without losing what you’ve started.