Secure Sensu

As with any piece of software, it is critical to minimize any attack surface the software exposes. Sensu is no different.

This reference describes the components you need to secure to make Sensu production-ready, including etcd peer communication, the Sensu API and web UI, and Sensu agent-to-server communication. It also describes agent mutual transport layer security (mTLS) authentication, which is required for secrets management.

Before you can secure Sensu, you must generate the certificates you will need. After you generate certificates, follow this reference to secure Sensu for production.

Secure etcd peer communication

To properly secure etcd communication, replace the default configuration option values in your backend store configuration in /etc/sensu/backend.yml as follows:

1. Replace the placeholder with the path to your certificate and key for the etcd-cert-file and etcd-key-file to secure client communication:

   etcd-cert-file: "/etc/sensu/tls/backend-1.example.com.pem"
   etcd-key-file: "/etc/sensu/tls/backend-1-key.example.com.pem"

2. Replace the placeholder with the path to your certificate and key for the etcd-peer-cert-file and etcd-peer-key-file to secure cluster communication:

   etcd-peer-cert-file: "/etc/sensu/tls/backend-1.example.com.pem"
   etcd-peer-key-file: "/etc/sensu/tls/backend-1-key.example.com.pem"

3. Replace the placeholder with the path to your ca.pem certificate for the etcd-trusted-ca-file and etcd-peer-trusted-ca-file to secure communication with the etcd client server and between etcd cluster members:

   etcd-trusted-ca-file: "/etc/sensu/tls/ca.pem"
   etcd-peer-trusted-ca-file: "/etc/sensu/tls/ca.pem"

4. Add non-default values for etcd-listen-client-urls, etcd-listen-peer-urls, and etcd-initial-advertise-peer-urls:

   etcd-listen-client-urls: "https://localhost:2379"
   etcd-listen-peer-urls: "https://localhost:2380"
   etcd-advertise-client-urls: "https://localhost:2379"
   etcd-initial-advertise-peer-urls: "https://localhost:2380"

   NOTE: If you are securing a cluster, use your backend node IP address instead of localhost in the non-default values for etcd-listen-client-urls, etcd-listen-peer-urls, and etcd-initial-advertise-peer-urls.

5. Set etcd-client-cert-auth and etcd-peer-client-cert-auth to true to ensure that etcd only allows connections from clients and peers that present a valid, trusted certificate:

   etcd-client-cert-auth: "true"
   etcd-peer-client-cert-auth: "true"

   Because etcd does not require authentication by default, you must set the etcd-client-cert-auth and etcd-peer-client-cert-auth configuration options to true to secure Sensu’s embedded etcd datastore against unauthorized access.

Secure the Sensu agent API, HTTP API, and web UI

The Sensu Go agent API, HTTP API, and web UI use a common stanza in /etc/sensu/backend.yml to provide the certificate, key, and CA file needed to provide secure communication.

Configure the following backend secure sockets layer (SSL) attributes in /etc/sensu/backend.yml:

1. Replace the placeholders with the paths to your CA root, backend certificate, and backend key files for the trusted-ca-file, cert-file, and key-file configuration options:

   trusted-ca-file: "/etc/sensu/tls/ca.pem"
   cert-file: "/etc/sensu/tls/backend-1.example.com.pem"
   key-file: "/etc/sensu/tls/backend-1-key.example.com.pem"

2. Set the insecure-skip-tls-verify configuration option to false:

   insecure-skip-tls-verify: false

3. When you provide these cert-file and key-file configuration options, the agent WebSocket API and HTTP API will serve requests over SSL/TLS (https). For this reason, you must also specify https:// schema for the api-url configuration option for backend API configuration:

   api-url: "https://localhost:8080"

Restart the sensu-backend service:

sudo systemctl restart sensu-backend

After you restart the sensu-backend service, the configuration options will load and you will able to access the web UI at https://localhost:3000. Configuring these options will also ensure that agents can communicate securely.

Specify a separate web UI certificate and key

You can use the same certificates and keys to secure etcd, the HTTP API, and the web UI. However, if you prefer, you can use a separate certificate and key for the web UI (for example, a commercially purchased certificate and key).

To do this, add the dashboard-cert-file and dashboard-key-file configuration options for backend SSL configuration in /etc/sensu/backend.yml:

dashboard-cert-file: "/etc/sensu/tls/separate-webui-cert.pem"
dashboard-key-file: "/etc/sensu/tls/separate-webui-key.pem"

Secure Sensu agent-to-server communication

By default, an agent uses the insecure ws:// transport. Here’s an example for agent configuration in /etc/sensu/agent.yml:

backend-url:
  - "ws://127.0.0.1:8081"

To use WebSocket over SSL/TLS (wss), change the backend-url value to the wss:// schema in /etc/sensu/agent.yml:

backend-url:
  - "wss://127.0.0.1:8081"

The agent will connect to Sensu backends over wss. Remember, if you change the configuration to wss, plaintext communication will not be possible.

You can also provide a trusted CA root certificate file as part of the agent configuration (named ca.pem in the example in Generate certificates). If you will start the agent via sensu-agent start, pass the --trusted-ca-file flag with the start command. Otherwise, include the trusted-ca-file configuration option in the agent configuration in /etc/sensu/agent.yml:

trusted-ca-file: "/etc/sensu/tls/ca.pem"

Optional: Configure Sensu agent mTLS authentication

By default, Sensu agents require username and password authentication to communicate with Sensu backends. For Sensu’s default user credentials and details about configuring Sensu role-based access control (RBAC), read the RBAC reference.

Alternatively, Sensu agents can use mTLS for authenticating to the backend WebSocket transport. When agent mTLS authentication is enabled, agents do not need to send password credentials to backends when they connect. To use secrets management, Sensu agents must be secured with mTLS. In addition, when using mTLS authentication, agents do not require an explicit user in Sensu. Sensu agents default to authenticating as the agent user and using permissions granted to the system:agents group by the system:agents cluster role and cluster role binding.

You can still bind agents to a specific user when the system:agents group is problematic. For this use case, create a user that matches the Common Name (CN) of the agent’s certificate.

For example, if you have a certificate named client.pem, you can run the following command to view the certificate’s CN with openssl:

openssl x509 -in client.pem -text -noout

The response should be similar to this example:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            37:57:7b:04:1d:67:63:7b:ff:ae:39:19:5b:55:57:80:41:3c:ec:ff
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = CA
        Validity
            Not Before: Sep 26 18:58:00 2019 GMT
            Not After : Sep 24 18:58:00 2024 GMT
        Subject: CN = client
...

The Subject: field indicates the certificate’s CN is client, so to bind the agent to a particular user in Sensu, create a user called client.

To enable agent mTLS authentication:

1. Create and distribute a new Certificate Authority (CA) root certificate and new agent and backend certificates and keys according to the Generate certificates guide.

2. Add the following configuration options and values to the backend configuration /etc/sensu/backend.yml:

   agent-auth-cert-file: "/etc/sensu/tls/backend-1.example.com.pem"
   agent-auth-key-file: "/etc/sensu/tls/backend-1-key.example.com.pem"
   agent-auth-trusted-ca-file: "/etc/sensu/tls/ca.pem"

3. Add the following configuration options and values to the agent configuration in /etc/sensu/agent.yml:

   cert-file: "/etc/sensu/tls/agent.pem"
   key-file: "/etc/sensu/tls/agent-key.pem"
   trusted-ca-file: "/etc/sensu/tls/ca.pem"

You can use certificates for authentication that are distinct from other communication channels used by Sensu, like etcd or the API. However, deployments can also use the same certificates and keys for etcd peer and client communication, the HTTP API, and agent authentication without issues.

Certificate revocation check

The Sensu backend checks certificate revocation list (CRL) and Online Certificate Status Protocol (OCSP) endpoints for agent mTLS, etcd client, and etcd peer connections whose remote sides present X.509 certificates that provide CRL and OCSP revocation information.

Optional: Configure Sensu for FIPS compliance

Sensu provides a Linux amd64 OpenSSL-linked build that supports the Federal Information Processing Standard (FIPS) for Federal Risk and Authorization Management Program (FedRAMP) compliance.

The Sensu build with FIPS-mode configuration options is linked with the FIPS 140-2 validated cryptographic library. Sensu builds comply with the FIPS-mode kernel option to enforce FIPS systemwide in Red Hat Enterprise Linux (RHEL). Contact Sensu to request the build with FIPS support.

Sensu backends and agents will work on systems with FIPS kernel mode if the require-fips configuration option is set to true in the backend and agent configuration files. Sensu backends and agents that have require-fips enabled will not work on systems without FIPS kernel mode.

Sensu backends on systems with FIPS kernel mode will work with PostgreSQL on systems with FIPS kernel mode. For PostgreSQL on systems without FIPS kernel mode, Sensu backends with FIPS kernel mode will work as long as the PostgreSQL system supports FIPS-compliant ciphers/cipher suites.

Sensu agents and sensuctl on systems with and without FIPS kernel mode can connect to Sensu backends on systems with FIPS kernel mode.

Configuration example for embedded etcd

To configure the Sensu backend for FIPS mode with embedded etcd, update the backend configuration file at /etc/sensu/backend.yml to use the following settings:

# fips configuration
require-fips: true

# etcd configuration
etcd-listen-client-urls: "https://localhost:2379"
etcd-listen-peer-urls: "https://localhost:2380"
etcd-advertise-client-urls: "https://localhost:2379"
etcd-initial-advertise-peer-urls: "https://localhost:2380"

# etcd client tls configuration
etcd-client-cert-auth: "true"
etcd-trusted-ca-file: "/etc/sensu/tls/ca.pem"
etcd-cert-file: "/etc/sensu/tls/centos-7-fips-1-backend.pem"
etcd-key-file: "/etc/sensu/tls/centos-7-fips-1-backend-key.pem"

# etcd peer tls configuration
etcd-peer-client-cert-auth: "true"
etcd-peer-trusted-ca-file: "/etc/sensu/tls/ca.pem"
etcd-peer-cert-file: "/etc/sensu/tls/centos-7-fips-1-backend.pem"
etcd-peer-key-file: "/etc/sensu/tls/centos-7-fips-1-backend-key.pem"

# api configuration
api-url: "https://localhost:8080"

# api tls configuration
insecure-skip-tls-verify: false
trusted-ca-file: "/etc/sensu/tls/ca.pem"
cert-file: "/etc/sensu/tls/centos-7-fips-1-backend.pem"
key-file: "/etc/sensu/tls/centos-7-fips-1-backend-key.pem"

Configuration example for external etcd

To configure the Sensu backend for FIPS mode with external etcd, update the backend configuration file at /etc/sensu/backend.yml to use the following settings:

# fips configuration
require-fips: true

# etcd configuration
etcd-trusted-ca-file: "/etc/sensu/tls/ca.pem"
etcd-cert-file: "/etc/sensu/tls/centos-7-fips-1-backend.pem"
etcd-key-file: "/etc/sensu/tls/centos-7-fips-1-backend-key.pem"
etcd-client-urls: "https://localhost:2379"
no-embed-etcd: true

# api configuration
api-url: "https://localhost:8080"

# api tls configuration
insecure-skip-tls-verify: false
trusted-ca-file: "/etc/sensu/tls/ca.pem"
cert-file: "/etc/sensu/tls/centos-7-fips-1-backend.pem"
key-file: "/etc/sensu/tls/centos-7-fips-1-backend-key.pem"

Use the following settings in your etcd configuration:

name: "centos-7-fips-1"
data-dir: "/var/lib/etcd-external"
auto-compaction-mode: "revision"
auto-compaction-retention: "2"

# cluster config
initial-cluster-token: "sup3rs3cr3t"
initial-cluster: "centos-7-fips-1=https://centos-7-fips-1:2380"
initial-cluster-state: "new"

# etcd configuration
listen-client-urls: "https://localhost:2379"
listen-peer-urls: "https://localhost:2380"
advertise-client-urls: "https://localhost:2379"
initial-advertise-peer-urls: "https://localhost:2380"

# etcd client tls configuration
client-transport-security:
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/tls/ca.pem
  cert-file: /etc/etcd/tls/centos-7-fips-1-backend.pem
  key-file: /etc/etcd/tls/centos-7-fips-1-backend-key.pem
  auto-tls: false

# etcd peer tls configuration
peer-transport-security:
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/tls/ca.pem
  cert-file: /etc/etcd/tls/centos-7-fips-1-backend.pem
  key-file: /etc/etcd/tls/centos-7-fips-1-backend-key.pem
  auto-tls: false

Next step: Run a Sensu cluster

Well done! Your Sensu installation should now be secured with TLS. The last step before you deploy Sensu is to set up a Sensu cluster.