Use secrets management in Sensu

Sensu’s secrets management allows you to avoid exposing secrets like usernames, passwords, and access keys in your Sensu configuration. In this guide, you’ll learn how to use Sensu’s Env secrets provider or HashiCorp Vault as your external secrets provider and authenticate without exposing your secrets. You’ll set up your PagerDuty Integration Key as a secret, create a PagerDuty handler definition that requires the secret, and configure a pipeline that includes the PagerDuty handler. Your Sensu backend can then execute the pipeline with any check.

To follow this guide, you’ll need to install the Sensu backend, have at least one Sensu agent running, and install and configure sensuctl.

Secrets are configured via secrets resources. A secret resource definition refers to the secrets provider (Env or VaultProvider) and an ID (the named secret to fetch from the secrets provider).

This guide only covers the handler use case, but you can use secrets management in handler, mutator, and check execution. When a check configuration references a secret, the Sensu backend will only transmit the check’s execution requests to agents that are connected via mutually authenticated transport layer security (mTLS)-encrypted WebSockets.

The secret included in your Sensu handler will be exposed to Sensu services at runtime as an environment variable. Sensu only exposes secrets to Sensu services like environment variables and automatically redacts secrets from all logs, the API, and the web UI.

Retrieve your PagerDuty Integration Key

The example in this guide uses the PagerDuty Integration Key as a secret and a PagerDuty handler definition that requires the secret.

Here’s how to find your Integration Key in PagerDuty so you can set it up as your secret:

1. Log in to your PagerDuty account.
2. In the Services drop-down menu, select Service Directory.
3. Enter the name of your Sensu service in the search field.
4. Click to select your Sensu service from the list of search results.
5. Click the Integrations tab.
6. Click the drop-down arrow for the Events API. The Integration Key is listed in the second field.

Make a note of your Integration Key — you’ll need it to create your backend environment variable or HashiCorp Vault secret.

Use Env for secrets management

The Sensu Go commercial distribution includes a secrets provider, Env, that exposes secrets from environment variables on your Sensu backend nodes. The Env secrets provider is automatically created with an empty spec when you start your Sensu backend.

Create your backend environment variable

To use the Env secrets provider, add your secret as a backend environment variable.

First, make sure you have created the files you need to store backend environment variables.

Then, run the following code, replacing INTEGRATION_KEY with your PagerDuty Integration Key:

echo 'SENSU_PAGERDUTY_KEY=INTEGRATION_KEY' | sudo tee -a /etc/default/sensu-backend

echo 'SENSU_PAGERDUTY_KEY=INTEGRATION_KEY' | sudo tee -a /etc/sysconfig/sensu-backend

Restart the sensu-backend:

sudo systemctl restart sensu-backend

This configures the SENSU_PAGERDUTY_KEY environment variable to your PagerDuty Integration Key in the context of the sensu-backend process.

Create your Env secret

Now you’ll use sensuctl create to create your secret. This code creates a secret named pagerduty_key that refers to the environment variable ID SENSU_PAGERDUTY_KEY. Run:

cat << EOF | sensuctl create
---
type: Secret
api_version: secrets/v1
metadata:
  name: pagerduty_key
spec:
  id: SENSU_PAGERDUTY_KEY
  provider: env
EOF

cat << EOF | sensuctl create
{
  "type": "Secret",
  "api_version": "secrets/v1",
  "metadata": {
    "name": "pagerduty_key"
  },
  "spec": {
    "id": "SENSU_PAGERDUTY_KEY",
    "provider": "env"
  }
}
EOF

You can securely pass your PagerDuty Integration Key in Sensu checks, handlers, and mutators by referring to the pagerduty_key secret. Skip to the add a handler section, where you’ll use your pagerduty_key secret in your handler definition.

Use HashiCorp Vault for secrets management

This section explains how to use HashiCorp Vault as your external secrets provider to authenticate via the HashiCorp Vault integration’s token auth method or TLS certificate auth method.

Configure your Vault authentication method (token or TLS)

If you use HashiCorp Vault as your external secrets provider, you can authenticate via the HashiCorp Vault integration’s token or transport layer security (TLS) certificate authentication method.

Vault token authentication

Follow the steps in this section to use HashiCorp Vault as your external secrets provider to authenticate with the HashiCorp Vault integration’s token auth method.

Retrieve your Vault root token

To retrieve your Vault root token:

1. Download and install the Vault edition for your operating system.
2. Open a terminal window and run vault server -dev.

The command output includes a Root Token line. Find this line in your command output and copy the Root Token value. You will use it next to create your Vault secrets provider.

Leave the Vault dev server running. Because you aren’t using TLS, you will need to set VAULT_ADDR=http://127.0.0.1:8200 in your shell environment.

Create your Vault secrets provider

Use sensuctl create to create your secrets provider, vault. In the code below, replace <root_token> with the Root Token value for your Vault dev server. Then, run:

cat << EOF | sensuctl create
---
type: VaultProvider
api_version: secrets/v1
metadata:
  name: vault
spec:
  client:
    address: http://localhost:8200
    token: <root_token>
    version: v2
    tls: null
    max_retries: 2
    timeout: 20s
    rate_limiter:
      limit: 10
      burst: 100
EOF

cat << EOF | sensuctl create
{
  "type": "VaultProvider",
  "api_version": "secrets/v1",
  "metadata": {
    "name": "vault"
  },
  "spec": {
    "client": {
      "address": "http://localhost:8200",
      "token": "<root_token>",
      "version": "v2",
      "tls": null,
      "max_retries": 2,
      "timeout": "20s",
      "rate_limiter": {
        "limit": 10,
        "burst": 100
      }
    }
  }
}
EOF

To continue, skip ahead to create your Vault secret.

Vault TLS certificate authentication

This section explains how use HashiCorp Vault as your external secrets provider to authenticate with the HashiCorp Vault integration’s TLS certificate auth method.

First, in your Vault, enable and configure certificate authentication. For example, your Vault might be configured for certificate authentication like this:

vault write auth/cert/certs/sensu-backend \
    display_name=sensu-backend \
    policies=sensu-backend-policy \
    certificate=@sensu-backend-vault.pem \
    ttl=3600

Second, configure your VaultProvider in Sensu:

---
type: VaultProvider
api_version: secrets/v1
metadata:
  name: vault
spec:
  client:
    address: https://vault.example.com:8200
    version: v2
    tls:
      ca_cert: /path/to/your/ca.pem
      client_cert: /etc/sensu/ssl/sensu-backend-vault.pem
      client_key: /etc/sensu/ssl/sensu-backend-vault-key.pem
      cname: sensu-backend.example.com
    max_retries: 2
    timeout: 20s
    rate_limiter:
      limit: 10
      burst: 100

{
  "type": "VaultProvider",
  "api_version": "secrets/v1",
  "metadata": {
    "name": "vault"
  },
  "spec": {
    "client": {
      "address": "https://vault.example.com:8200",
      "version": "v2",
      "tls": {
        "ca_cert": "/path/to/your/ca.pem",
        "client_cert": "/etc/sensu/ssl/sensu-backend-vault.pem",
        "client_key": "/etc/sensu/ssl/sensu-backend-vault-key.pem",
        "cname": "sensu-backend.example.com"
      },
      "max_retries": 2,
      "timeout": "20s",
      "rate_limiter": {
        "limit": 10,
        "burst": 100
      }
    }
  }
}

The certificate you specify for tls.client_cert should be the same certificate you configured in your Vault for certificate authentication.

Next, create your Vault secret.

Create your Vault secret

First, retrieve your PagerDuty Integration Key (the secret you will set up in Vault).

Next, open a new terminal and run vault kv put secret/pagerduty key=<integration_key>. Replace <integration_key> with your PagerDuty Integration Key. This writes your secret into Vault.

In this example, the name of the secret is pagerduty. The pagerduty secret contains a key, and you specified that the key value is your PagerDuty Integration Key.

Run vault kv get secret/pagerduty to view the secret you just set up.

Use sensuctl create to create your vault secret:

cat << EOF | sensuctl create
---
type: Secret
api_version: secrets/v1
metadata:
  name: pagerduty_key
spec:
  id: secret/pagerduty#key
  provider: vault
EOF

cat << EOF | sensuctl create
{
  "type": "Secret",
  "api_version": "secrets/v1",
  "metadata": {
    "name": "pagerduty_key"
  },
  "spec": {
    "id": "secret/pagerduty#key",
    "provider": "vault"
  }
}
EOF

Now you can securely pass your PagerDuty Integration Key in the handlers, and mutators by referring to the pagerduty_key secret. In the add a handler section, you’ll use your pagerduty_key secret in your handler definition.

Add a handler

Register the sensu/sensu-pagerduty-handler dynamic runtime asset

To begin, register the sensu/sensu-pagerduty-handler dynamic runtime asset with sensuctl asset add:

sensuctl asset add sensu/sensu-pagerduty-handler:2.2.0 -r pagerduty-handler

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: pagerduty-handler.

Run sensuctl asset list --format yaml to confirm that the dynamic runtime asset is ready to use.

With this handler, Sensu can trigger and resolve PagerDuty incidents. However, you still need to add your secret to the handler spec so that it requires your backend to request secrets from your secrets provider.

Add your secret to the handler spec

To create a handler definition that uses your pagerduty_key secret, run:

cat << EOF | sensuctl create
---
api_version: core/v2
type: Handler
metadata:
  name: pagerduty
spec:
  type: pipe
  command: pagerduty-handler --token $PD_TOKEN
  secrets:
  - name: PD_TOKEN
    secret: pagerduty_key
  runtime_assets:
  - pagerduty-handler
  timeout: 10
EOF

cat << EOF | sensuctl create
{
  "api_version": "core/v2",
  "type": "Handler",
  "metadata": {
    "name": "pagerduty"
  },
  "spec": {
    "type": "pipe",
    "command": "pagerduty-handler --token $PD_TOKEN",
    "secrets": [
      {
        "name": "PD_TOKEN",
        "secret": "pagerduty_key"
      }
    ],
    "runtime_assets": [
      "pagerduty-handler"
    ],
    "timeout": 10
  }
}
EOF

Configure a pipeline

Now that your handler is set up and Sensu can create incidents in PagerDuty, you can configure a pipeline to start receiving alerts based on the events your checks create. A single pipeline workflow can include one or more filters, one mutator, and one handler.

In this case, the pipeline will include the built-in is_incident event filter and the pagerduty handler you created in the previous step. You can add this pipeline to any check to receive a PagerDuty alert for every warning (1) or critical (2) event the check generates, as well as for resolution events.

To create the pipeline, run:

cat << EOF | sensuctl create
---
type: Pipeline
api_version: core/v2
metadata:
  name: incident_alerts
spec:
  workflows:
  - name: pagerduty_incidents
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    handler:
      name: pagerduty
      type: Handler
      api_version: core/v2
EOF

cat << EOF | sensuctl create
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "incident_alerts"
  },
  "spec": {
    "workflows": [
      {
        "name": "pagerduty_incidents",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "pagerduty",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}
EOF

To automate this workflow, include the incident_alerts pipeline in any Sensu check definition in the check’s pipelines attribute. When you list a pipeline in a check definition, all the observability events that the check produces will be processed according to the pipeline’s workflows.

Next steps

Add your pipeline to any check to start receiving PagerDuty alerts based on observability event data. Read Send PagerDuty alerts with Sensu for an example that shows how to edit a check definition to add a pipeline.

Read the secrets or secrets providers reference for in-depth secrets management documentation.