Pipelines reference

IMPORTANT: The pipelines described on this page are different from the resources you can create and manage with the enterprise/pipeline/v1 API. The enterprise/pipeline/v1 API allows you to create and manage resources that can only be used in pipelines rather than pipelines themselves.

Read the Sumo Logic metrics handlers reference and TCP stream handlers reference for more information about enterprise pipeline resources.

Sensu executes pipelines during the process stage of the observability pipeline.

Pipelines are Sensu resources composed of observation event processing workflows made up of filters, mutators, and handlers. Instead of specifying filters and mutators in handler definitions, you can specify all three in a single pipeline workflow.

To use a pipeline, list it in a check definition’s pipelines array. All the observability events that the check produces will be processed according to the pipeline’s workflows.

Pipelines can replace handler sets and handler stacks. We recommend migrating your existing handler sets and stacks to pipeline workflows.

NOTE: To use pipelines, upgrade your agents to Sensu Go 6.5.0.

Pipeline example

This example shows a pipeline resource definition that includes event filters, a mutator, and a handler:

---
type: Pipeline
api_version: core/v2
metadata:
  name: incident_alerts
spec:
  workflows:
  - name: labeled_email_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: not_silenced
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    mutator:
      name: add_labels
      type: Mutator
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "incident_alerts"
  },
  "spec": {
    "workflows": [
      {
        "name": "labeled_email_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "not_silenced",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "mutator": {
          "name": "add_labels",
          "type": "Mutator",
          "api_version": "core/v2"
        },
        "handler": {
          "name": "email",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}

To use this pipeline in a check, list it in the check’s pipelines array. For example:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: incident_pipelines
spec:
  command: collect.sh
  interval: 10
  publish: true
  subscriptions:
  - system
  pipelines:
  - type: Pipeline
    api_version: core/v2
    name: incident_alerts
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "incident_pipelines"
  },
  "spec": {
    "command": "collect.sh",
    "interval": 10,
    "publish": true,
    "subscriptions": [
      "system"
    ],
    "pipelines": [
      {
        "type": "Pipeline",
        "api_version": "core/v2",
        "name": "incident_alerts"
      }
    ]
  }
}

Workflows

The workflows attribute is an array of event processing workflows that Sensu will apply for events produced by any check that references the pipeline.

Workflows do not have to include an event filter or mutator, but they must specify at least one handler.

Workflows can include more than one event filter. If a workflow has more than one filter, Sensu applies the filters in a series, starting with the filter that is listed first.

You can use your existing event filters, mutators, and handlers in pipeline workflows. Pipelines ignore any filters and mutators specified in handler definitions, so you do not need to remove them to use your existing handlers — just make sure to define the event filters and mutators you want to use in the pipeline workflow.

Pipelines with multiple workflows

Pipelines can include more than one workflow.

In this example, the pipeline includes labeled_email_alerts and slack_alerts workflows:

---
type: Pipeline
api_version: core/v2
metadata:
  name: incident_alerts
spec:
  workflows:
  - name: labeled_email_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: not_silenced
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    mutator:
      name: add_labels
      type: Mutator
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
  - name: slack_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: not_silenced
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    handler:
      name: slack
      type: Handler
      api_version: core/v2
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "incident_alerts"
  },
  "spec": {
    "workflows": [
      {
        "name": "labeled_email_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "not_silenced",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "mutator": {
          "name": "add_labels",
          "type": "Mutator",
          "api_version": "core/v2"
        },
        "handler": {
          "name": "email",
          "type": "Handler",
          "api_version": "core/v2"
        }
      },
      {
        "name": "slack_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "not_silenced",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "slack",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}

All events from checks that specify this pipeline will be processed with both workflows, in series, starting with the workflow that is listed first in the resource definition.

Read Route alerts with event filters for another pipeline example that includes multiple workflows for contact-based routing.

Pipeline specification

Top-level attributes

api_version
description Top-level attribute that specifies the Sensu API group and version. For pipelines in this version of Sensu, the api_version should always be core/v2.
required Required for pipeline definitions in wrapped-json or yaml format for use with sensuctl create.
type String
example 
api_version: core/v2
{
  "api_version": "core/v2"
}
metadata
description Top-level collection of metadata about the pipeline that includes name, namespace, and created_by as well as custom labels and annotations. The metadata map is always at the top level of the pipeline definition. This means that in wrapped-json and yaml formats, the metadata scope occurs outside the spec scope. Read metadata attributes for details.
required Required for pipeline definitions in wrapped-json or yaml format for use with sensuctl create.
type Map of key-value pairs
example 
metadata:
  name: incident_alerts
  namespace: default
  created_by: admin
  labels:
    region: us-west-1
  annotations:
    slack-channel: "#incidents"
{
  "metadata": {
    "name": "incident_alerts",
    "namespace": "default",
    "created_by": "admin",
    "labels": {
      "region": "us-west-1"
    },
    "annotations": {
      "slack-channel": "#incidents"
    }
  }
}
spec
description Top-level map that includes the pipeline spec attributes.
required Required for pipeline definitions in wrapped-json or yaml format for use with sensuctl create.
type Map of key-value pairs
example 
spec:
  workflows:
  - name: labeled_email_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: not_silenced
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    mutator:
      name: add_labels
      type: Mutator
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
{
  "spec": {
    "workflows": [
      {
        "name": "labeled_email_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "not_silenced",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "mutator": {
          "name": "add_labels",
          "type": "Mutator",
          "api_version": "core/v2"
        },
        "handler": {
          "name": "email",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}
type
description Top-level attribute that specifies the sensuctl create resource type. Pipelines should always be type Pipeline.
required Required for pipeline definitions in wrapped-json or yaml format for use with sensuctl create.
type String
example 
type: Pipeline
{
  "type": "Pipeline"
}

Metadata attributes

annotations
description Non-identifying metadata to include with observation event data that you can access with event filters. You can use annotations to add data that’s meaningful to people or external tools that interact with Sensu.

In contrast to labels, you cannot use annotations in API response filtering, sensuctl response filtering, or web UI views.
required false
type Map of key-value pairs. Keys and values can be any valid UTF-8 string.
default null
example 
annotations:
  managed-by: ops
  slack-channel: "#incidents"
{
  "annotations": {
    "managed-by": "ops",
    "slack-channel": "#incidents"
  }
}
created_by
description Username of the Sensu user who created the pipeline or last updated the handler. Sensu automatically populates the created_by field when the pipeline is created or updated.
required false
type String
example 
created_by: admin
{
  "created_by": "admin"
}
labels
description Custom attributes to include with observation event data that you can use for response and web UI view filtering.

If you include labels in your event data, you can filter API responses, sensuctl responses, and web UI views based on them. In other words, labels allow you to create meaningful groupings for your data.

Limit labels to metadata you need to use for response filtering. For complex, non-identifying metadata that you will not need to use in response filtering, use annotations rather than labels.
required false
type Map of key-value pairs. Keys can contain only letters, numbers, and underscores and must start with a letter. Values can be any valid UTF-8 string.
default null
example 
labels:
  environment: production
  region: us-west-1
{
  "labels": {
    "environment": "production",
    "region": "us-west-1"
  }
}
name
description Unique string used to identify the pipeline. Pipeline names cannot contain special characters or spaces (validated with Go regex \A[\w\.\-]+\z). Each pipeline must have a unique name within its namespace.
required true
type String
example 
name: incident_alerts
{
  "name": "incident_alerts"
}
namespace
description Sensu RBAC namespace that the pipeline belongs to.
required false
type String
default default
example 
namespace: default
{
  "namespace": "default"
}

Spec attributes

workflows
description Array of workflows (by names) to use when filtering, mutating, and handling observability events with a pipeline. Each array item must be a string. Read workflows attributes for details.
required false
type Array
example 
workflows:
  - name: labeled_email_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: not_silenced
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    mutator:
      name: add_labels
      type: Mutator
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
{
  "workflows": [
    {
      "name": "labeled_email_alerts",
      "filters": [
        {
          "name": "is_incident",
          "type": "EventFilter",
          "api_version": "core/v2"
        },
        {
          "name": "not_silenced",
          "type": "EventFilter",
          "api_version": "core/v2"
        },
        {
          "name": "state_change_only",
          "type": "EventFilter",
          "api_version": "core/v2"
        }
      ],
      "mutator": {
        "name": "add_labels",
        "type": "Mutator",
        "api_version": "core/v2"
      },
      "handler": {
        "name": "email",
        "type": "Handler",
        "api_version": "core/v2"
      }
    }
  ]
}

Workflows attributes

filters
description Reference for the Sensu event filters to use when filtering events for the pipeline. Each pipeline workflow can reference more than one event filter. If a workflow has more than one filter, Sensu applies the filters in a series, starting with the filter that is listed first. Read filters attributes for details.
required false
type Map of key-value pairs
default null
example 
filters:
- name: is_incident
  type: EventFilter
  api_version: core/v2
- name: not_silenced
  type: EventFilter
  api_version: core/v2
- name: state_change_only
  type: EventFilter
  api_version: core/v2
{
  "filters": [
    {
      "name": "is_incident",
      "type": "EventFilter",
      "api_version": "core/v2"
    },
    {
      "name": "not_silenced",
      "type": "EventFilter",
      "api_version": "core/v2"
    },
    {
      "name": "state_change_only",
      "type": "EventFilter",
      "api_version": "core/v2"
    }
  ]
}

handler
description Reference for the Sensu handler to use for event processing in the workflow. Each pipeline workflow must reference one handler. Pipelines ignore any filters and mutators specified in handler definitions. Read handler attributes for details.
required true
type Map of key-value pairs
example 
handler:
  name: email
  type: Handler
  api_version: core/v2
{
  "handler": {
    "name": "email",
    "type": "Handler",
    "api_version": "core/v2"
  }
}
mutator
description Reference for the Sensu mutator to use to mutate event data for the workflow. Each pipeline workflow can reference only one mutator. Read mutator attributes for details.
required false
type Map of key-value pairs
default null
example 
mutator:
  name: add_labels
  type: Mutator
  api_version: core/v2
{
  "mutator": {
    "name": "add_labels",
    "type": "Mutator",
    "api_version": "core/v2"
  }
}

Filters attributes

api_version
description The Sensu API group and version for the event filter. For event filters in this version of Sensu, the api_version should always be core/v2.
required true
type String
default null
example 
api_version: core/v2
{
  "api_version": "core/v2"
}
name
description Name of the Sensu event filter to use for the workflow. You can use the built-in event filters, as well as your existing event filters, in pipeline workflows.
required true
type String
default null
example 
name: is_incident
{
  "name": "is_incident"
}
type
description The sensuctl create resource type for the event filter. Event filters should always be type EventFilter.
required true
type String
default null
example 
type: EventFilter
{
 "type": "EventFilter"
}

Handler attributes

api_version
description The Sensu API group and version for the handler.
required true
type String
allowed values core/v2 for a pipe handler, TCP or UDP handler, or handler set

pipeline/v1 for a TCP stream handler or Sumo Logic metrics handler
default null
example 
api_version: core/v2
{
  "api_version": "core/v2"
}
name
description Name of the Sensu handler to use for the workflow. You can use your existing handlers in pipeline workflows — pipelines ignore any filters and mutators specified in handler definitions.
required true
type String
default null
example 
name: email
{
  "name": "email"
}
type
description The sensuctl create resource type for the handler.
required true
type String
allowed values Handler for a pipe handler, TCP or UDP handler, or handler set

TCPStreamHandler for a TCP stream handler

SumoLogicMetricsHandler for a Sumo Logic metrics handler
default null
example 
type: Handler
{
 "type": "Handler"
}

Mutator attributes

api_version
description The Sensu API group and version for the mutator. For mutators in this version of Sensu, the api_version should always be core/v2.
required true
type String
default null
example 
api_version: core/v2
{
  "api_version": "core/v2"
}
name
description Name of the Sensu mutator to use for the workflow. You can use your existing mutators in pipeline workflows.
required true
type String
default null
example 
name: add_labels
{
  "name": "add_labels"
}
type
description The sensuctl create resource type for the mutator. Mutators should always be type Mutator.
required true
type String
default null
example 
type: Mutator
{
 "type": "Mutator"
}