Send email alerts with a pipeline

Pipelines are Sensu resources composed of observation event processing workflows that include filters, mutators, and handlers. You can use pipelines to send email alerts, create or resolve incidents (in PagerDuty, for example), or store metrics in a time-series database like InfluxDB.

When you are using Sensu in production, events will come from a check or metric you configure. For this guide, you will create an ad hoc event that you can trigger manually to test your email handler.

Your backend will execute a pipeline with a handler that sends notifications to the email address you specify. The pipeline will also include an event filter to make sure you only receive a notification when your event represents a status change.

Requirements

To follow this guide, install the Sensu backend, make sure at least one Sensu agent is running, and configure sensuctl to connect to the backend as the admin user.

Add the email handler dynamic runtime asset

Dynamic runtime assets are shareable, reusable packages that help you deploy Sensu plugins. In this guide, you’ll use the sensu/sensu-email-handler dynamic runtime asset to power an email handler.

Use the following sensuctl example to register the sensu/sensu-email-handler dynamic runtime asset:

sensuctl asset add sensu/sensu-email-handler:1.2.2 -r email-handler

The response will confirm that the asset was added:

fetching bonsai asset: sensu/sensu-email-handler:1.2.2
added asset: sensu/sensu-email-handler:1.2.2

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["email-handler"].

The -r (rename) flag allows you to specify a shorter name for the dynamic runtime asset (in this case, email-handler).

To confirm that the handler dynamic runtime asset was added correctly, run:

sensuctl asset list

The list should include the email-handler dynamic runtime asset. For a detailed list of everything related to the asset that Sensu added automatically, run:

sensuctl asset info email-handler

The sensu/sensu-email-handler dynamic runtime asset includes the sensu-email-handler command, which you will use when you create the email handler definition later in this guide.

Add an event filter

Event filters allow you to fine-tune how your events are handled and reduce alert fatigue. In this guide, your event filter will send notifications only when your event’s state changes (for example, for any change between 0 OK, 1 warning, and 2 critical).

Here’s an overview of how the state_change_only filter will work:

• If your event status changes from 0 to 1, you will receive one email notification for the change to warning status.
• If your event status stays at 1 for the next hour, you will not receive repeated email notifications during that hour.
• If your event status changes to 2 after 1 hour at 1, you will receive one email notification for the change from warning to critical status.
• If your event status fluctuates between 0, 1, and 2 for the next hour, you will receive one email notification each time the status changes.

To create the event filter, run:

cat << EOF | sensuctl create
---
type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: state_change_only
spec:
  action: allow
  expressions:
  - event.check.occurrences == 1
  runtime_assets: []
EOF

cat << EOF | sensuctl create
{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "annotations": null,
    "labels": null,
    "name": "state_change_only"
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.occurrences == 1"
    ],
    "runtime_assets": [

    ]
  }
}
EOF

Create the email handler definition

After you add an event filter, create the email handler definition to specify the email address where the handler will send notifications. In the handler definition’s command value, you’ll need to change a few things:

• <sender@example.com>: Replace with the email address you want to use to send email alerts.
• <recipient@example.com>: Replace with the email address you want to receive email alerts.
• <smtp_server@example.com>: Replace with the hostname of your SMTP server.
• <username>: Replace with your SMTP username, typically your email address.
• <password>: Replace with your SMTP password, typically the same as your email password.

After you update the command with your email, server, username, and password values in the example below, run the updated code to create the email handler definition:

cat << EOF | sensuctl create
---
api_version: core/v2
type: Handler
metadata:
  name: email
spec:
  type: pipe
  command: sensu-email-handler -f <sender@example.com> -t <recipient@example.com> -s <smtp_server@example.com> -u username -p password
  timeout: 10
  runtime_assets:
  - email-handler
EOF

cat << EOF | sensuctl create
{
  "api_version": "core/v2",
  "type": "Handler",
  "metadata": {
    "name": "email"
  },
  "spec": {
    "type": "pipe",
    "command": "sensu-email-handler -f <sender@example.com> -t <recipient@example.com> -s <smtp_server@example.com> -u username -p password",
    "timeout": 10,
    "runtime_assets": [
      "email-handler"
    ]
  }
}
EOF

Create a pipeline

With your event filter and handler configured, you can build a pipeline workflow. A single pipeline workflow can include one or more filters, one mutator, and one handler.

In this case, the pipeline includes your state_change_only event filter and email handler, as well as two built-in event filters, is_incident and not_silenced. These two built-in filters are included in every Sensu backend installation, so you don’t have to create them. The is_incident and not_silenced event filters ensure that you receive alerts for unsilenced events with only warning (1) or critical (2) status:

To create the pipeline, run:

cat << EOF | sensuctl create
---
type: Pipeline
api_version: core/v2
metadata:
  name: alerts_pipeline
spec:
  workflows:
  - name: email_alerts
    filters:
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: not_silenced
      type: EventFilter
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
EOF

cat << EOF | sensuctl create
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "alerts_pipeline"
  },
  "spec": {
    "workflows": [
      {
        "name": "email_alerts",
        "filters": [
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "not_silenced",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "email",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}
EOF

Before your pipeline can send alerts to your email, you need an event that generates the alerts. In the final step, you will create an ad hoc event that you can trigger manually.

Create and trigger an ad hoc event

To create an ad hoc event, first use sensuctl env to set up environment variables. The environment variables will provide the required Sensu API access token credential for the Sensu API:

eval $(sensuctl env)

Verify that the SENSU_ACCESS_TOKEN environment variable is set by echoing its value:

echo $SENSU_ACCESS_TOKEN

The response will list the SENSU_ACCESS_TOKEN value.

With the environment variables set, you can use the Sensu API to create your ad hoc observability event.

This event outputs the message “Everything is OK.” when it occurs:

curl -sS -H 'Content-Type: application/json' \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "Everything is OK.",
    "status": 0
  }
}' \
http://localhost:8080/api/core/v2/namespaces/default/events

As configured, the event status is 0 (OK). Now it’s time to trigger an event and view the results!

To generate a status change event, use the update event endpoint to create a 1 (warning) event. Run:

curl -sS -X PUT \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-H 'Content-Type: application/json' \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "This is a warning.",
    "status": 1
  },
  "pipelines": [
    {
      "type": "Pipeline",
      "api_version": "core/v2",
      "name": "alerts_pipeline"
    }
  ]
}' \
http://localhost:8080/api/core/v2/namespaces/default/events/server01/server-health

Check your email — you should receive a message from Sensu!

Create another event with status set to 0. Run:

curl -sS -X PUT \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-H 'Content-Type: application/json' \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "Everything is OK.",
    "status": 0
  },
  "pipelines": [
    {
      "type": "Pipeline",
      "api_version": "core/v2",
      "name": "alerts_pipeline"
    }
  ]
}' \
http://localhost:8080/api/core/v2/namespaces/default/events/server01/server-health

You should receive another email because the event status changed to 0 (OK).

Next steps

Now that you know how to apply a pipeline to a check and take action on events, reuse this email pipeline with the check_cpu check from our Monitor server resources guide. Check out Route alerts with event filters for a complex pipeline example that includes several workflows with different event filters and handlers.

The sensu/sensu-email-handler dynamic runtime asset makes it possible to add a template that provides context for your email notifications. The email template functionality uses tokens to populate the values provided by the event, and you can use HTML to format the email.