Generate certificates for your Sensu installation
This guide explains how to generate the certificates you need to secure a Sensu cluster and its agents.
When deploying Sensu for use outside of a local development environment, you should secure it using transport layer security (TLS). TLS uses encryption to provide security for communication between Sensu backends and agents as well as communication between human operators and the Sensu backend, such as web UI or sensuctl access.
Because reconfiguring an existing Sensu deployment from cleartext to TLS can be time-consuming, we recommend that you configure TLS for your backend from the very beginning.
TLS is also required to use some of Sensu’s commercial features, like secrets management and mutual TLS authentication (mTLS).
Prerequisites
To use this guide, you must have already installed Sensu on:
- One backend system or three backend systems that you plan to cluster together.
- One or more agents.
Public key infrastructure (PKI)
To use TLS, you must either possess existing public key infrastructure (PKI) or generate your own Certificate Authority (CA) for issuing certificates.
This guide describes how to set up a minimal CA and generate the certificates you need to secure Sensu communications for a clustered backend and agents.
If your organization has existing PKI for certificate issuance, you can adapt the suggestions in this guide to your organization’s PKI. Recommended practices for deploying and maintaining production PKI can be complex and case-specific, so they are not included in the scope of this guide.
Issue certificates
Use a CA certificate and key to generate certificates and keys to use with Sensu backends and agents.
This guide uses the CloudFlare cfssl toolkit to generate a CA and self-signed certificates from that CA.
The examples assume that you’ll install the certificates and keys in the /etc/sensu/tls
directory.
Install TLS
The CloudFlare cfssl toolkit is released as a collection of command-line tools.
These tools only need to be installed on one system to generate your CA and issue certificates.
You may install the toolkit on your laptop or workstation and store the files there for safekeeping or install the toolkit on one of the systems where you’ll run the Sensu backend. The example in this guide installs cfssl on a Linux system.
-
Download the cfssl executable:
sudo curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64 -o /usr/local/bin/cfssl
-
Download the cfssljson executable:
sudo curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64 -o /usr/local/bin/cfssljson
-
Install the cfssl and cfssljson executables in /usr/local/bin:
sudo chmod +x /usr/local/bin/cfssl*
-
Verify the cfssl executable is version 1.4.1 and runtime go1.12.12:
cfssl version
-
Verify the cfssljson executable is version 1.4.1 and runtime go1.12.12:
cfssljson -version
Create a Certificate Authority (CA)
Follow these steps to create a CA with cfssl and cfssljson:
-
Create
/etc/sensu/tls
(which does not exist by default):mkdir -p /etc/sensu/tls
-
Navigate to the new
/etc/sensu/tls
directory:cd /etc/sensu/tls
-
Create the CA:
echo '{"CN":"Sensu Test CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
-
Define signing parameters and profiles (the agent profile provides the “client auth” usage required for mTLS):
echo '{"signing":{"default":{"expiry":"17520h","usages":["signing","key encipherment","client auth"]},"profiles":{"backend":{"usages":["signing","key encipherment","server auth","client auth"],"expiry":"4320h"},"agent":{"usages":["signing","key encipherment","client auth"],"expiry":"4320h"}}}}' > ca-config.json
NOTE: We suggest a 6-month expiry duration for security, but you can use any duration you prefer when you define the
expiry
attribute value in the signing parameters.
You should now have a directory at /etc/sensu/tls
that contains the following files:
filename | description |
---|---|
ca.pem |
CA root certificate. Required for all systems running the Sensu backend or agent. The agent and backend use ca.pem to validate server certificates at connection time. |
ca-key.pem |
CA root certificate private key. |
ca-config.json |
CA signing parameters and profiles. Not used by Sensu. |
ca.csr |
Certificate signing request for the CA root certificate. Not used by Sensu. |
Generate backend cluster certificates
Now that you’ve generated a CA, you will use it to generate certificates and keys for each backend server (etcd peer).
For each backend server, document the IP addresses and hostnames to use in backend and agent communications.
During initial configuration of a cluster of Sensu backends, you must describe every member of the cluster with a URL passed as the value of the etcd-initial-cluster
parameter.
In issuing certificates for cluster members, the IP address or hostname used in these URLs must be represented in either the Common Name (CN) or Subject Alternative Name (SAN) records in the certificate.
NOTE: As of Go 1.15, certificates must include their CN as an SAN field. Follow the instructions in this guide to make sure your certificates’ SAN fields include their CNs.
This guide assumes a scenario with three backend members that are reachable via a 10.0.0.x
IP address, a fully qualified name (for example, backend-1.example.com
), and an unqualified name (for example, backend-1
):
Unqualified name |
IP address | Fully qualified domain name (FQDN) |
Additional names |
---|---|---|---|
backend-1 | 10.0.0.1 | backend-1.example.com | localhost, 127.0.0.1 |
backend-2 | 10.0.0.2 | backend-2.example.com | localhost, 127.0.0.1 |
backend-3 | 10.0.0.3 | backend-3.example.com | localhost, 127.0.0.1 |
The additional names for localhost and 127.0.0.1 are added here for convenience and are not strictly required.
Use these name and address details to create two *.pem
files and one *.csr
file for each backend.
- The values provided for the ADDRESS variable will be used to populate the certificate’s SAN records. For systems with multiple hostnames and IP addresses, add each to the comma-delimited value of the ADDRESS variable.
- The value provided for the NAME variable will be used to populate the certificate’s CN record.
It will also be used in the names for the
*.pem
and*.csr
files.
For example, to create certificate and key files for the three backends:
backend-1
export ADDRESS=localhost,127.0.0.1,10.0.0.1,backend-1
export NAME=backend-1.example.com
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -profile="backend" -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
backend-2
export ADDRESS=localhost,127.0.0.1,10.0.0.2,backend-2
export NAME=backend-2.example.com
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -profile="backend" -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
backend-3
export ADDRESS=localhost,127.0.0.1,10.0.0.3,backend-3
export NAME=backend-3.example.com
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -profile="backend" -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
The /etc/sensu/tls
directory should now include three files for each backend, in addition to the four original CA files:
filename | description | required on backend? |
---|---|---|
backend-*.pem |
Backend server certificate | |
backend-*-key.pem |
Backend server private key | |
backend-*.csr |
Certificate signing request |
In our example with three backends, the directory listing for /etc/sensu/tls
would include 13 files:
/etc/sensu/tls/
├── backend-1-key.example.com.pem
├── backend-1.example.com.pem
├── backend-1.example.com.csr
├── backend-2-key.example.com.pem
├── backend-2.example.com.pem
├── backend-2.example.com.csr
├── backend-3-key.example.com.pem
├── backend-3.example.com.pem
├── backend-3.example.com.csr
├── ca.pem
├── ca-key.pem
├── ca-config.json
└── ca.csr
WARNING: If you are not setting up agent mTLS authentication, delete the ca-key.pem
file from the /etc/sensu/tls
directory.
The ca-key.pem
file contains sensitive information and is no longer needed unless you are setting up agent mTLS authentication.
To make sure the backend files in /etc/sensu/tls
are accessible only by the sensu
user, run:
chown sensu /etc/sensu/tls/*.pem
And:
chmod 400 /etc/sensu/tls/*.pem
Generate agent certificate
NOTE: Agent certificates are only required for agent mTLS authentication. If you are not configuring mTLS for Sensu agents, you do not need to generate agent certificates.
Now you will generate a certificate that agents can use to connect to the Sensu backend. Sensu’s commercial distribution offers support for authenticating agents via TLS certificates instead of a username and password.
For this certificate, you only need to specify a CN (here, agent
) — you don’t need to specify an address.
You will create the files agent.pem
, agent-key.pem
, and agent.csr
:
export NAME=agent
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="" -profile=agent - | cfssljson -bare $NAME
The /etc/sensu/tls
directory should now include a set of files for use by Sensu agents:
filename | description | required on agent? |
---|---|---|
agent.pem |
Agent certificate | |
agent-key.pem |
Agent private key | |
agent.csr |
Certificate signing request |
WARNING: Before you continue, delete the ca-key.pem
file from the /etc/sensu/tls
directory.
This file contains sensitive information and is no longer needed.
To continue the example with three backends, the directory listing for /etc/sensu/tls
will include 15 files after deleting the ca-key.pem
file:
/etc/sensu/tls/
├── agent-key.pem
├── agent.pem
├── agent.csr
├── backend-1-key.example.com.pem
├── backend-1.example.com.pem
├── backend-1.example.com.csr
├── backend-2-key.example.com.pem
├── backend-2.example.com.pem
├── backend-2.example.com.csr
├── backend-3-key.example.com.pem
├── backend-3.example.com.pem
├── backend-3.example.com.csr
├── ca.pem
├── ca-config.json
└── ca.csr
To make sure the agent /etc/sensu/tls
files are accessible only by the sensu
user, run:
chown sensu /etc/sensu/tls/*.pem
And:
chmod 400 /etc/sensu/tls/*.pem
Install CA certificates
Before you install the CA certificates, make sure that the /etc/sensu/tls
directory does not contain the ca-key.pem
file.
The ca-key.pem
file contains sensitive information that is no longer needed, so you should delete it.
Also, make sure that /etc/sensu/tls
includes the CA root certificate and key, as well as a certificate and key for each backend and agent you are securing.
We recommend installing the CA root certificate in the trust store of both your Sensu systems and those systems used by operators to manage Sensu. Installing the CA certificate in the trust store for these systems makes it easier to connect via web UI or sensuctl without being prompted to accept certificates signed by your self-generated CA.
chmod 644 /etc/sensu/tls/ca.pem
chown root /etc/sensu/tls/ca.pem
sudo apt-get install ca-certificates -y
sudo ln -sfv /etc/sensu/tls/ca.pem /usr/local/share/ca-certificates/sensu-ca.crt
sudo update-ca-certificates
chmod 644 /etc/sensu/tls/ca.pem
chown root /etc/sensu/tls/ca.pem
sudo yum install -y ca-certificates
sudo update-ca-trust force-enable
sudo ln -s /etc/sensu/tls/ca.pem /etc/pki/ca-trust/source/anchors/sensu-ca.pem
sudo update-ca-trust
Import the root CA certificate on the Mac.
Double-click the root CA certificate to open it in Keychain Access.
The root CA certificate appears in login.
Copy the root CA certificate to System to ensure that it is trusted by all users and local system processes.
Open the root CA certificate, expand Trust, select Use System Defaults, and save your changes.
Reopen the root CA certificate, expand Trust, select Always Trust, and save your changes.
Delete the root CA certificate from login.
Press Windows+R to open the Run dialog.
Type "MMC" (without quotation marks) in the Run dialog and press Enter to open the MMC console.
In the MMC console, expand the Certificates (Local Computer) node and navigate to Trusted Root Certification Authorities > Certificates.
Right-click the Trusted Root Certification Authorities > Certificates folder and select All Tasks > Import to open the Certificate Import dialog.
In the Certificate Import dialog, click Next and browse to the location where the root CA certificate is stored.
Select the root CA certificate file and click Open.
Click Next, click Next, and click Finish.
Renew self-generated certificates
To keep your Sensu deployment running smoothly, renew your self-generated certificates before they expire. Depending on how your certificates are configured, one backend certificate may expire before the others or all three backend certificates may expire at the same time. The agent certificate also expires.
This section explains how to find certificate expiration dates, confirm whether certificates have already expired, and renew certificates.
Find certificate expiration dates
Use this check to find certificate expiration dates so you can renew certificates before they expire and avoid observability interruptions.
Before you run the check, replace <cert-name>.pem
in the command with the name of the certificate you want to check (for example, backend-1.example.com.pem
).
---
type: CheckConfig
api_version: core/v2
metadata:
name: expired_certs
spec:
command: openssl x509 -noout -enddate -in <cert-name>.pem
subscriptions:
- system
publish: true
{
"type": "CheckConfig",
"api_version": "core/v2",
"metadata": {
"name": "expired_certs"
},
"spec": {
"command": "openssl x509 -noout -enddate -in <cert-name>.pem",
"subscriptions": [
"system"
],
"publish": true
}
}
The check output will be in the format notAfter=Month Day HH:MM:SS Year Timezone
.
For example:
notAfter=Jul 3 22:23:50 2021 GMT
Add a handler to send the check output as a notification or to a log file.
Identify expired certificates
The following sensuctl cluster health
response indicates that one backend certificate is expired:
Error: GET "/health": Get https://localhost:8080/health?timeout=3: x509: certificate has expired or is not yet valid
The log for the expired backend will be similar to this example:
backend-1.example.com | {"component":"etcd","level":"warning","msg":"health check for peer a95ca1cdb0b1fcc3 could not connect: remote error: tls: bad certificate (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-22T20:40:54Z"}
backend-1.example.com | {"component":"etcd","level":"warning","msg":"health check for peer a95ca1cdb0b1fcc3 could not connect: remote error: tls: bad certificate (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-22T20:40:54Z"}
If you restart the cluster with one expired backend certificate, the sensuctl cluster health
response will include an error:
Error: GET "/health": failed to request new refresh token; client returned 'Post https://localhost:8080/auth/token: EOF'
When all three backend certificates are expired, the log will be similar to this example:
backend-1.example.com | {"component":"etcd","level":"warning","msg":"health check for peer a95ca1cdb0b1fcc3 could not connect: x509: certificate has expired or is not yet valid (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-25T17:49:53Z"}
backend-2.example.com | {"component":"etcd","level":"warning","msg":"health check for peer 4cc36e198efb22e8 could not connect: x509: certificate has expired or is not yet valid (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-25T17:49:16Z"}
backend-3.example.com | {"component":"etcd","level":"warning","msg":"health check for peer 8425a7b2d2ee8597 could not connect: x509: certificate has expired or is not yet valid (prober \"ROUND_TRIPPER_RAFT_MESSAGE\")","pkg":"rafthttp","time":"2021-06-25T17:49:16Z"}
If you restart the cluster with three expired backend certificates, the sensuctl cluster health
response will include an error:
Error: GET "/health": Get https://127.0.0.1:8080/health?timeout=3: EOF
The following sensuctl cluster health
response helps confirm that all three backend certificates are expired, together with the log warning and restart error examples:
=== Etcd Cluster ID: 45c04eab9efc0d11
ID Name Error Healthy
────────────────── ──────────────────────── ─────────────────────────── ─────────
a95ca1cdb0b1fcc3 backend-1.example.com context deadline exceeded false
8425a7b2d2ee8597 backend-2.example.com context deadline exceeded false
4cc36e198efb22e8 backend-3.example.com context deadline exceeded false
An expired agent certificate does not cause any errors or log messages to indicate the expiration. Use the certificate expiration check to find the agent certificate expiration date.
Renew certificates
To renew your certificates, whether they expired or not, follow the steps to create a CA, generate backend certificates, or generate an agent certificate. The new certificate will override the existing certificate.
After you save the new certificates, restart each backend:
sudo systemctl start sensu-backend
Next step: Secure Sensu
Now that you have generated the required certificates, follow Secure Sensu to make your Sensu installation production-ready.