Backend reference

Example Sensu backend configuration file (download)

The Sensu backend is a service that manages check requests and observability data. Every Sensu backend includes an integrated structure for scheduling checks using subscriptions, an event processing pipeline that applies event filters, mutators, handlers, and pipelines, an embedded etcd datastore for storing configuration and state, and the Sensu API, Sensu web UI, and sensuctl command line tool.

The Sensu backend is available for Debian- and RHEL-family distributions of Linux. For these operating systems, the Sensu backend uses the Bourne shell (sh) for the execution environment.

Read the installation guide to install the backend.

Initialization

For a new installation, the backend database must be initialized by providing a username and password for the user to be granted administrative privileges. Although initialization is required for every new installation, the implementation differs depending on your method of installation:

• If you are using Docker, you can use environment variables to override the default admin username (admin) and password (P@ssw0rd!) during step 2 of the backend installation process.
• If you are using a Debian- or RHEL-family distribution, you must specify admin credentials during step 3 of the backend installation process. Sensu does not apply default admin credentials for Debian- or RHEL-family installations.

The initialization step bootstraps the first admin user account for your Sensu installation. This first account will be granted the cluster admin role.

Docker initialization

For Docker installations, set administrator credentials with environment variables when you configure and start the backend as shown below. Replace <username> and <password> with the username and password you want to use:

docker run -v /var/lib/sensu:/var/lib/sensu \
-d --name sensu-backend \
-p 3000:3000 -p 8080:8080 -p 8081:8081 \
-e SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=<username> \
-e SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=<password> \
sensu/sensu:latest \
sensu-backend start --state-dir /var/lib/sensu/sensu-backend --log-level debug

---
version: "3"
services:
  sensu-backend:
    ports:
    - 3000:3000
    - 8080:8080
    - 8081:8081
    volumes:
    - "sensu-backend-data:/var/lib/sensu/sensu-backend/etcd"
    command: "sensu-backend start --state-dir /var/lib/sensu/sensu-backend --log-level debug"
    environment:
    - SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=<username>
    - SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=<password>
    image: sensu/sensu:latest

volumes:
  sensu-backend-data:
    driver: local

If you did not use environment variables to override the default admin credentials in step 2 of the backend installation process, we recommend changing your default admin password as soon as you have installed sensuctl.

Debian or RHEL family initialization

For Debian- or RHEL-family distributions, set administrator credentials with environment variables at initialization as shown below.

To initialize with your username and password, replace <username> and <password> with the username and password you want to use:

export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=<username>
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=<password>
sensu-backend init

Add API key for initialization

Add an API key when you initialize the backend to make automated cluster setup and deployment more straightforward. An API key is a persistent UUID that maps to a stored Sensu username.

If you supply an API key via sensu-backend init, you do not need to configure sensuctl. Instead, you can execute sensuctl commands to manage resources immediately after initializing a cluster by providing the --api-key and --api-url flags with their correct values in your sensuctl commands.

To initialize with an API key in addition to username and password, set your administrator credentials as follows. Replace <api_key> with the API key you want to use:

docker run -v /var/lib/sensu:/var/lib/sensu \
-d --name sensu-backend \
-p 3000:3000 -p 8080:8080 -p 8081:8081 \
-e SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=<username> \
-e SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=<password> \
-e SENSU_BACKEND_CLUSTER_ADMIN_API_KEY=<api_key> \
sensu/sensu:latest \
sensu-backend start --state-dir /var/lib/sensu/sensu-backend --log-level debug

---
version: "3"
services:
  sensu-backend:
    ports:
    - 3000:3000
    - 8080:8080
    - 8081:8081
    volumes:
    - "sensu-backend-data:/var/lib/sensu/sensu-backend/etcd"
    command: "sensu-backend start --state-dir /var/lib/sensu/sensu-backend --log-level debug"
    environment:
    - SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=<username>
    - SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=<password>
    - SENSU_BACKEND_CLUSTER_ADMIN_API_KEY=<api_key>
    image: sensu/sensu:latest

volumes:
  sensu-backend-data:
    driver: local

export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=<username>
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=<password>
export SENSU_BACKEND_CLUSTER_ADMIN_API_KEY=<api_key>
sensu-backend init

Initialize in interactive mode

You can also run the sensu-backend init command in interactive mode:

sensu-backend init --interactive

You will receive prompts for username, password, and API key in interactive mode. Provide your username and password to complete initialization. The API key is optional — press return to skip it.

Cluster Admin Username: <username>
Cluster Admin Password: <password>
Retype Cluster Admin Password: <password>
Cluster Admin API Key: <api_key>

Initialization flags

To view available initialization flags:

sensu-backend init --help

The response will list command information and configuration flags for sensu-backend init:

Usage:
  sensu-backend init [flags]

General Flags:
      --cluster-admin-api-key string    cluster admin API key
      --cluster-admin-password string   cluster admin password
      --cluster-admin-username string   cluster admin username
  -c, --config-file string              path to sensu-backend config file (default "/etc/sensu/backend.yml")
  -h, --help                            help for init
      --ignore-already-initialized      exit 0 if the cluster has already been initialized
      --interactive                     interactive mode
      --timeout string                  duration to wait before a connection attempt to etcd is considered failed (must be >= 1s) (default "5s")
      --wait                            continuously retry to establish a connection to etcd until it is successful

Store Flags:
      --etcd-advertise-client-urls strings   list of this member's client URLs to advertise to clients (default [http://localhost:2379])
      --etcd-cert-file string                path to the client server TLS cert file
      --etcd-cipher-suites strings           list of ciphers to use for etcd TLS configuration
      --etcd-client-cert-auth                enable client cert authentication
      --etcd-client-urls string              client URLs to use when operating as an etcd client
      --etcd-key-file string                 path to the client server TLS key file
      --etcd-max-request-bytes uint          maximum etcd request size in bytes (use with caution) (default 1572864)
      --etcd-trusted-ca-file string          path to the client server TLS trusted CA cert file

For more information about the initialization store flags, read Datastore and cluster configuration and Advanced configuration options.

sensu-backend init --ignore-already-initialized

sensu-backend init --timeout 30s

sensu-backend init --wait

Backend transport

The Sensu backend listens for agent communications via WebSocket transport. By default, this transport operates on port 8081. The agent subscriptions are used to determine which check execution requests the backend publishes via the transport. Sensu agents locally execute checks as requested by the backend and publish check results back to the transport to be processed.

Sensu agents authenticate to the Sensu backend via transport by either built-in username and password authentication or mutual transport layer security (mTLS) authentication.

To secure the WebSocket transport, first generate the certificates you will need to set up transport layer security (TLS). Then, secure Sensu by configuring either TLS or mTLS to make Sensu production-ready.

Read the Sensu architecture overview for a diagram that includes the WebSocket transport.

Certificate bundles or chains

The Sensu backend supports all types of certificate bundles (or chains) as long as the server (or leaf) certificate is the first certificate in the bundle. This is because the Go standard library assumes that the first certificate listed in the PEM file is the server certificate — the certificate that the program will use to show its own identity.

If you send the server certificate alone instead of sending the whole bundle with the server certificate first, you will receive a certificate not signed by trusted authority error. You must present the whole chain to the remote so it can determine whether it trusts the server certificate through the chain.

Certificate revocation check

The Sensu backend checks certificate revocation list (CRL) and Online Certificate Status Protocol (OCSP) endpoints for mutual transport layer security (mTLS), etcd client, and etcd peer connections whose remote sides present X.509 certificates that provide CRL and OCSP revocation information.

Startup and backend entities

When a backend starts up, Sensu automatically checks for a sensu-system namespace (and creates the namespace if it doesn’t exist). Then, Sensu checks the sensu-system namespace for an existing entity named after the backend’s local hostname.

• If there is no corresponding entity, Sensu creates a new entity with entity_class: backend and populates the entity’s system information.
• If there is a corresponding entity, Sensu does nothing further to the existing entity.

Once the backend entity is created, the backend uses its own entity to report cluster state errors. Read backend entities in the entities reference for more information and an example backend entity definition.

Synchronize time between agents and the backend

System clocks between agents and the backend should be synchronized to a central NTP server. If system time is out of sync, it may cause issues with keepalive, metric, and check alerts.

Backend clusters

You can run the backend as a standalone service, but running a cluster of backends makes Sensu more highly available, reliable, and durable. Sensu backend clusters build on the etcd clustering system. Clustering lets you synchronize data between backends and get the benefits of a highly available configuration.

To configure a cluster, read Run a Sensu cluster and review the datastore configuration options.

Create event pipelines

Sensu backend event pipelines process observation data and execute event filters, mutators, handlers, and pipelines. These resources are powerful tools to automate your monitoring workflows.

Read the event filter, mutator, handler, and pipeline references to learn more about these Sensu resources. Read guides like Reduce alert fatigue with event filters and Send Slack alerts with handlers for usage examples.

Schedule checks

The backend is responsible for storing check definitions and scheduling check requests. Check scheduling is subscription-based: the backend sends check requests to subscriptions, where they’re picked up by subscribing agents.

For information about creating and managing checks, read the checks reference and the guides Monitor server resources with checks and Collect metrics with checks.

Backend configuration options

Backend configuration is customizable. This section describes each configuration option in more detail, including examples for each configuration method.

You can customize backend configuration with the backend configuration file, command line flag arguments, or environment variables.

To view configuration information for the sensu-backend start command, run:

sensu-backend start --help

The response will list configuration options as command line flags for sensu-backend start:

start the sensu backend

Usage:
  sensu-backend start [flags]

General Flags:
      --agent-auth-cert-file string               TLS certificate in PEM format for agent certificate authentication
      --agent-auth-crl-urls strings               URLs of CRLs for agent certificate authentication
      --agent-auth-key-file string                TLS certificate key in PEM format for agent certificate authentication
      --agent-auth-trusted-ca-file string         TLS CA certificate bundle in PEM format for agent certificate authentication
      --agent-burst-limit int                     agent connections maximum burst size
      --agent-host string                         agent listener host (default "[::]")
      --agent-serve-wait-time duration            wait time before accepting agent connections on startup
      --agent-port int                            agent listener port (default 8081)
      --agent-rate-limit int                      agent connections maximum rate limit
      --agent-write-timeout int                   timeout in seconds for agent writes (default 15)
      --annotations stringToString                entity annotations map (default [])
      --api-listen-address string                 address to listen on for api traffic (default "[::]:8080")
      --api-serve-wait-time duration              wait time before serving API requests on startup
      --api-request-limit int                     maximum API request body size, in bytes (default 512000)
      --api-url string                            url of the api to connect to (default "http://localhost:8080")
      --api-write-timeout                         maximum duration before timing out writes of responses
      --assets-burst-limit int                    asset fetch burst limit (default 100)
      --assets-rate-limit float                   maximum number of assets fetched per second
      --cache-dir string                          path to store cached data (default "/var/cache/sensu/sensu-backend")
      --cert-file string                          TLS certificate in PEM format
  -c, --config-file string                        path to sensu-backend config file (default "/etc/sensu/backend.yml")
      --dashboard-cert-file string                dashboard TLS certificate in PEM format
      --dashboard-host string                     dashboard listener host (default "[::]")
      --dashboard-key-file string                 dashboard TLS certificate key in PEM format
      --dashboard-port int                        dashboard listener port (default 3000)
      --dashboard-write-timeout                   maximum duration before timing out writes of responses
      --debug                                     enable debugging and profiling features
      --deregistration-handler string             default deregistration handler
      --disable-platform-metrics                  disable platform metrics logging
      --event-log-buffer-size int                 buffer size of the event logger (default 100000)
      --event-log-buffer-wait string              full buffer wait time (default "10ms")
      --event-log-file string                     path to the event log file
      --event-log-parallel-encoders               used to indicate parallel encoders should be used for event logging
      --eventd-buffer-size int                    number of incoming events that can be buffered (default 100)
      --eventd-workers int                        number of workers spawned for processing incoming events (default 100)
  -h, --help                                      help for start
      --insecure-skip-tls-verify                  skip TLS verification (not recommended!)
      --jwt-private-key-file string               path to the PEM-encoded private key to use to sign JWTs
      --jwt-public-key-file string                path to the PEM-encoded public key to use to verify JWT signatures
      --keepalived-buffer-size int                number of incoming keepalives that can be buffered (default 100)
      --keepalived-workers int                    number of workers spawned for processing incoming keepalives (default 100)
      --key-file string                           TLS certificate key in PEM format
      --labels stringToString                     entity labels map (default [])
      --log-level string                          logging level [panic, fatal, error, warn, info, debug, trace] (default "warn")
      --metrics-refresh-interval string           Go duration value (e.g. 1h5m30s) that governs how often metrics are refreshed. (default "1m")
      --pipelined-buffer-size int                 number of events to handle that can be buffered (default 100)
      --pipelined-workers int                     number of workers spawned for handling events through the event pipeline (default 100)
      --platform-metrics-log-file string          platform metrics log file path
      --platform-metrics-logging-interval string  platform metrics logging interval
      --require-fips                              indicates whether fips support should be required in openssl  
      --trusted-ca-file string                    TLS CA certificate bundle in PEM format

Store Flags:
      --etcd-advertise-client-urls strings        list of this member's client URLs to advertise to clients (default [http://localhost:2379])
      --etcd-cert-file string                     path to the client server TLS cert file
      --etcd-cipher-suites strings                list of ciphers to use for etcd TLS configuration
      --etcd-client-cert-auth                     enable client cert authentication
      --etcd-client-urls string                   client URLs to use when operating as an etcd client
      --etcd-discovery string                     discovery URL used to bootstrap the cluster
      --etcd-discovery-srv string                 DNS SRV record used to bootstrap the cluster
      --etcd-election-timeout uint                time in ms a follower node will go without hearing a heartbeat before attempting to become leader itself (default 3000)
      --etcd-heartbeat-interval uint              interval in ms with which the etcd leader will notify followers that it is still the leader (default 300)
      --etcd-initial-advertise-peer-urls strings  list of this member's peer URLs to advertise to the rest of the cluster (default [http://127.0.0.1:2380])
      --etcd-initial-cluster string               initial cluster configuration for bootstrapping
      --etcd-initial-cluster-state string         initial cluster state ("new" or "existing") (default "new")
      --etcd-initial-cluster-token string         initial cluster token for the etcd cluster during bootstrap
      --etcd-key-file string                      path to the client server TLS key file
      --etcd-client-log-level string              etcd client logging level [panic, fatal, error, warn, info, debug] (default "error")
      --etcd-listen-client-urls strings           list of etcd client URLs to listen on (default [http://127.0.0.1:2379])
      --etcd-listen-peer-urls strings             list of URLs to listen on for peer traffic (default [http://127.0.0.1:2380])
      --etcd-log-level string                     etcd server logging level [panic, fatal, error, warn, info, debug]
      --etcd-max-request-bytes uint               maximum etcd request size in bytes (use with caution) (default 1572864)
      --etcd-name string                          name for this etcd node (default "default")
      --etcd-peer-cert-file string                path to the peer server TLS cert file
      --etcd-peer-client-cert-auth                enable peer client cert authentication
      --etcd-peer-key-file string                 path to the peer server TLS key file
      --etcd-peer-trusted-ca-file string          path to the peer server TLS trusted CA file
      --etcd-quota-backend-bytes int              maximum etcd database size in bytes (use with caution) (default 4294967296)
      --etcd-trusted-ca-file string               path to the client server TLS trusted CA cert file
      --etcd-unsafe-no-fsync                      disables fsync, unsafe, may cause data loss
      --no-embed-etcd                             don't embed etcd, use external etcd instead

The backend requires that the state-dir configuration option is set before starting. All other required flags have default values.

For more information about log configuration options, read Event logging and Platform metrics logging.

General configuration

sensu-backend start --annotations sensu.io/plugins/slack/config/webhook-url=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
sensu-backend start --annotations example-key="example value" --annotations example-key2="example value"

annotations:
  sensu.io/plugins/slack/config/webhook-url: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"

sensu-backend start --api-listen-address [::]:8080

api-listen-address: "[::]:8080"

sensu-backend start --api-request-limit 1024000

api-request-limit: 1024000

sensu-backend start --api-serve-wait-time 10s

api-serve-wait-time: 10s

sensu-backend start --api-url http://localhost:8080

api-url: "http://localhost:8080"

sensu-backend start --api-write-timeout 15s

api-write-timeout: 15s

sensu-backend start --assets-burst-limit 100

assets-burst-limit: 100

sensu-backend start --assets-rate-limit 1.39

assets-rate-limit: 1.39

sensu-backend start --cache-dir /var/cache/sensu-backend

cache-dir: "/var/cache/sensu-backend"

sensu-backend start --config-file /etc/sensu/backend.yml
sensu-backend start -c /etc/sensu/backend.yml

sensu-backend start --debug

debug: true

sensu-backend start --deregistration-handler deregister

deregistration-handler: "deregister"

sensu-backend start --labels security_zone=us-west-2a
sensu-backend start --labels example_key1="example value" example_key2="example value"

labels:
  security_zone: "us-west-2a"
  example_key1: "example value"
  example_key2: "example value"

sensu-backend start --log-level debug

log-level: "debug"

sensu-backend start --metrics-refresh-interval 10s

metrics-refresh-interval: 10s

sensu-backend start --state-dir /var/lib/sensu/sensu-backend
sensu-backend start -d /var/lib/sensu/sensu-backend

state-dir: "/var/lib/sensu/sensu-backend"

Agent communication configuration

sensu-backend start --agent-auth-cert-file /path/to/tls/backend-1.pem

agent-auth-cert-file: /path/to/tls/backend-1.pem

sensu-backend start --agent-auth-crl-urls http://localhost/CARoot.crl

agent-auth-crl-urls: http://localhost/CARoot.crl

sensu-backend start --agent-auth-key-file /path/to/tls/backend-1-key.pem

agent-auth-key-file: /path/to/tls/backend-1-key.pem

sensu-backend start --agent-auth-trusted-ca-file /path/to/tls/ca.pem

agent-auth-trusted-ca-file: /path/to/tls/ca.pem

sensu-backend start --agent-burst-limit 10

agent-burst-limit: 10

sensu-backend start --agent-host 127.0.0.1

agent-host: "127.0.0.1"

sensu-backend start --agent-serve-wait-time 10s

agent-serve-wait-time: 10s

sensu-backend start --agent-port 8081

agent-port: 8081

sensu-backend start --agent-rate-limit 10

agent-rate-limit: 10

Security configuration

sensu-backend start --cert-file /path/to/tls/backend-1.pem

cert-file: "/path/to/tls/backend-1.pem"

sensu-backend start --insecure-skip-tls-verify

insecure-skip-tls-verify: true

sensu-backend start --jwt-private-key-file /path/to/key/private.pem

jwt-private-key-file: /path/to/key/private.pem

sensu-backend start --jwt-public-key-file /path/to/key/public.pem

jwt-public-key-file: /path/to/key/public.pem

sensu-backend start --key-file /path/to/tls/backend-1-key.pem

key-file: "/path/to/tls/backend-1-key.pem"

sensu-backend start --require-fips

require-fips: true

sensu-backend start --require-openssl

require-openssl: true

sensu-backend start --trusted-ca-file /path/to/tls/ca.pem

trusted-ca-file: "/path/to/tls/ca.pem"

Web UI configuration

sensu-backend start --dashboard-cert-file /path/to/tls/separate-webui-cert.pem

dashboard-cert-file: "/path/to/tls/separate-webui-cert.pem"

sensu-backend start --dashboard-host 127.0.0.1

dashboard-host: "127.0.0.1"

sensu-backend start --dashboard-key-file /path/to/tls/separate-webui-key.pem

dashboard-key-file: "/path/to/tls/separate-webui-key.pem"

sensu-backend start --dashboard-port 3000

dashboard-port: 3000

sensu-backend start --dashboard-write-timeout 15s

dashboard-write-timeout: 15s

Datastore and cluster configuration

sensu-backend start --etcd-advertise-client-urls http://localhost:2378,http://localhost:2379
sensu-backend start --etcd-advertise-client-urls http://localhost:2378 --etcd-advertise-client-urls http://localhost:2379

etcd-advertise-client-urls:
  - http://localhost:2378
  - http://localhost:2379

sensu-backend start --etcd-cert-file /path/to/tls/backend-1.pem

etcd-cert-file: "/path/to/tls/backend-1.pem"

etcd-cipher-suites:
  - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
  - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

sensu-backend start --etcd-cipher-suites TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
sensu-backend start --etcd-cipher-suites TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 --etcd-cipher-suites TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

etcd-cipher-suites:
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

sensu-backend start --etcd-client-cert-auth

etcd-client-cert-auth: true

sensu-backend start --etcd-client-log-level error

etcd-client-log-level: "error"

sensu-backend start --etcd-client-urls 'https://10.0.0.1:2379 https://10.1.0.1:2379'
sensu-backend start --etcd-client-urls https://10.0.0.1:2379 --etcd-client-urls https://10.1.0.1:2379

etcd-client-urls:
  - https://10.0.0.1:2379
  - https://10.1.0.1:2379

sensu-backend start --etcd-discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de

etcd-discovery:
  - https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de

sensu-backend start --etcd-discovery-srv example.org

etcd-discovery-srv:
  - example.org

sensu-backend start --etcd-initial-advertise-peer-urls https://10.0.0.1:2380,https://10.1.0.1:2380
sensu-backend start --etcd-initial-advertise-peer-urls https://10.0.0.1:2380 --etcd-initial-advertise-peer-urls https://10.1.0.1:2380

etcd-initial-advertise-peer-urls:
  - https://10.0.0.1:2380
  - https://10.1.0.1:2380

sensu-backend start --etcd-initial-cluster backend-0=https://10.0.0.1:2380,backend-1=https://10.1.0.1:2380,backend-2=https://10.2.0.1:2380

etcd-initial-cluster: "backend-0=https://10.0.0.1:2380,backend-1=https://10.1.0.1:2380,backend-2=https://10.2.0.1:2380"

sensu-backend start --etcd-initial-cluster-state existing

etcd-initial-cluster-state: "existing"

sensu-backend start --etcd-initial-cluster-token unique_token_for_this_cluster

etcd-initial-cluster-token: "unique_token_for_this_cluster"

sensu-backend start --etcd-key-file /path/to/tls/backend-1-key.pem

etcd-key-file: "/path/to/tls/backend-1-key.pem"

sensu-backend start --etcd-listen-client-urls https://10.0.0.1:2379,https://10.1.0.1:2379
sensu-backend start --etcd-listen-client-urls https://10.0.0.1:2379 --etcd-listen-client-urls https://10.1.0.1:2379

etcd-listen-client-urls:
  - https://10.0.0.1:2379
  - https://10.1.0.1:2379

sensu-backend start --etcd-listen-peer-urls https://10.0.0.1:2380,https://10.1.0.1:2380
sensu-backend start --etcd-listen-peer-urls https://10.0.0.1:2380 --etcd-listen-peer-urls https://10.1.0.1:2380

etcd-listen-peer-urls:
  - https://10.0.0.1:2380
  - https://10.1.0.1:2380

sensu-backend start --etcd-log-level debug

etcd-log-level: "debug"

sensu-backend start --etcd-name backend-0

etcd-name: "backend-0"

sensu-backend start --etcd-peer-cert-file /path/to/tls/backend-1.pem

etcd-peer-cert-file: "/path/to/tls/backend-1.pem"

sensu-backend start --etcd-peer-client-cert-auth

etcd-peer-client-cert-auth: true

sensu-backend start --etcd-peer-key-file /path/to/tls/backend-1-key.pem

etcd-peer-key-file: "/path/to/tls/backend-1-key.pem"

sensu-backend start --etcd-peer-trusted-ca-file ./ca.pem

etcd-peer-trusted-ca-file: "./ca.pem"

sensu-backend start --etcd-trusted-ca-file ./ca.pem

etcd-trusted-ca-file: "./ca.pem"

sensu-backend start --etcd-unsafe-no-fsync

etcd-unsafe-no-fsync: true

sensu-backend start --no-embed-etcd

no-embed-etcd: true

Advanced configuration options

sensu-backend start --etcd-election-timeout 3000

etcd-election-timeout: 3000

sensu-backend start --etcd-heartbeat-interval 300

etcd-heartbeat-interval: 300

sensu-backend start --etcd-max-request-bytes 1572864

etcd-max-request-bytes: 1572864

sensu-backend start --etcd-quota-backend-bytes 4294967296

etcd-quota-backend-bytes: 4294967296

sensu-backend start --eventd-buffer-size 100

eventd-buffer-size: 100

sensu-backend start --eventd-workers 100

eventd-workers: 100

sensu-backend start --keepalived-buffer-size 100

keepalived-buffer-size: 100

sensu-backend start --keepalived-workers 100

keepalived-workers: 100

sensu-backend start --pipelined-buffer-size 100

pipelined-buffer-size: 100

sensu-backend start --pipelined-workers 100

pipelined-workers: 100

Backend configuration methods

Backend configuration file

You can customize the backend configuration in a .yml configuration file. The default backend configuration file path for Linux is /etc/sensu/backend.yml.

To use the backend.yml file to configure the backend, list the desired configuration attributes and values. Review the example Sensu backend configuration file for a complete example.

Configuration via command line flags or environment variables overrides any configuration specified in the backend configuration file. Read Create overrides to learn more.

Command line flags

You can customize the backend configuration with sensu-agent start command line flags.

To use command line flags, specify the desired configuration options and values along with the sensu-backend start command. For example:

sensu-backend start --deregistration-handler slack_deregister --log-level debug

Configuration via command line flags overrides attributes specified in a configuration file or with environment variables. Read Create overrides to learn more.

Environment variables

Instead of using a configuration file or command line flags, you can use environment variables to configure your Sensu backend. Each backend configuration option has an associated environment variable. You can also create your own environment variables, as long as you name them correctly and save them in the correct place. Here’s how.

1. Create the files from which the sensu-backend service configured by our supported packages will read environment variables:

   sudo touch /etc/default/sensu-backend

   sudo touch /etc/sysconfig/sensu-backend

2. Make sure the environment variable is named correctly.

   • To rename a configuration option you wish to specify as an environment variable, prepend SENSU_BACKEND_, convert dashes to underscores, and capitalize all letters. For example, the environment variable for the configuration option api-listen-address is SENSU_BACKEND_API_LISTEN_ADDRESS.

   • For a custom environment variable, you do not have to prepend SENSU_BACKEND. For example, TEST_VAR_1 is a valid custom environment variable name.

3. Add the environment variable to the environment file.

   For example, to create api-listen-address as an environment variable and set it to 192.168.100.20:8080:

   echo 'SENSU_BACKEND_API_LISTEN_ADDRESS=192.168.100.20:8080' | sudo tee -a /etc/default/sensu-backend

   echo 'SENSU_BACKEND_API_LISTEN_ADDRESS=192.168.100.20:8080' | sudo tee -a /etc/sysconfig/sensu-backend

4. Restart the sensu-backend service so these settings can take effect:

   sudo systemctl restart sensu-backend

   sudo systemctl restart sensu-backend

Format for label and annotation environment variables

To use labels and annotations as environment variables in your handler configurations, you must use a specific format when you create the label and annotation environment variables.

For example, to create the labels "region": "us-east-1" and "type": "website" as an environment variable:

echo 'BACKEND_LABELS='{"region": "us-east-1", "type": "website"}'' | sudo tee -a /etc/default/sensu-backend

echo 'BACKEND_LABELS='{"region": "us-east-1", "type": "website"}'' | sudo tee -a /etc/sysconfig/sensu-backend

To create the annotations "maintainer": "Team A" and "webhook-url": "https://hooks.slack.com/services/T0000/B00000/XXXXX" as an environment variable:

echo 'BACKEND_ANNOTATIONS='{"maintainer": "Team A", "webhook-url": "https://hooks.slack.com/services/T0000/B00000/XXXXX"}'' | sudo tee -a /etc/default/sensu-backend

echo 'BACKEND_ANNOTATIONS='{"maintainer": "Team A", "webhook-url": "https://hooks.slack.com/services/T0000/B00000/XXXXX"}'' | sudo tee -a /etc/sysconfig/sensu-backend

Use environment variables with the Sensu backend

Any environment variables you create in /etc/default/sensu-backend (Debian family) or /etc/sysconfig/sensu-backend (RHEL family) will be available to handlers executed by the Sensu backend.

For example, if you create a custom environment variable TEST_VARIABLE in your sensu-backend file, it will be available to use in your handler configurations as $TEST_VARIABLE. The following handler will print the TEST_VARIABLE value set in your sensu-backend file in /tmp/test.txt:

---
type: Handler
api_version: core/v2
metadata:
  name: print_test_var
spec:
  command: echo $TEST_VARIABLE >> ./tmp/test.txt
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "print_test_var"
  },
  "spec": {
    "command": "echo $TEST_VARIABLE >> ./tmp/test.txt",
    "timeout": 0,
    "type": "pipe"
  }
}

Create configuration overrides

Sensu has default settings and limits for certain configuration attributes, like the default log level. Depending on your environment and preferences, you may want to create overrides for these Sensu-specific defaults and limits.

You can create configuration overrides in several ways:

• Command line configuration flag arguments for sensu-backend start.
• Environment variables in /etc/default/sensu-backend (Debian family) or /etc/sysconfig/sensu-backend (RHEL family).
• Configuration settings in the backend.yml config file.

Sensu applies the following precedence to override settings:

1. Arguments passed to the Sensu backend via command line configuration flags.
2. Environment variables in /etc/default/sensu-backend (Debian family) or /etc/sysconfig/sensu-backend (RHEL family).
3. Configuration in the backend.yml config file.

For example, if you create overrides using all three methods, the command line configuration flag values will take precedence over the values you specify in /etc/default/sensu-backend or /etc/sysconfig/sensu-backend or the backend.yml config file.

Example override: Log level

The default log level for the Sensu backend is warn. To override the default and automatically apply a different log level for the backend, add the --log-level command line configuration flag when you start the Sensu backend. For example, to specify debug as the log level:

sensu-backend start --log-level debug

To configure an environment variable for the desired backend log level:

echo 'SENSU_BACKEND_LOG_LEVEL=debug' | sudo tee -a /etc/default/sensu-backend

echo 'SENSU_BACKEND_LOG_LEVEL=debug' | sudo tee -a /etc/sysconfig/sensu-backend

To configure the desired log level in the config file, add this line to backend.yml:

log-level: debug

Event logging

If you wish, you can log all Sensu event data to a file in JSON format. The Sensu event log file can be a reliable input source for your favorite data lake solution as well as a buffer for event data that you send to a database in case the database is unavailable.

Depending on the number and size of events, logging status and metrics events to a file can require intensive input/output (I/O) performance. Make sure you have adequate I/O capacity before using the event logging function.

Use these backend configuration options to customize event logging:

sensu-backend start --event-log-buffer-size 100000

event-log-buffer-size: 100000

sensu-backend start --event-log-buffer-wait 10ms

event-log-buffer-wait: 10ms

sensu-backend start --event-log-file /var/log/sensu/events.log

event-log-file: "/var/log/sensu/events.log"

sensu-backend start --event-log-parallel-encoders true

event-log-parallel-encoders: true

Log rotation

To manually rotate event logs, first rename (move) the current log file. Then, send the SIGHUP signal to the sensu-backend process so it creates a new log file and starts logging to it. Most Linux distributions include logrotate to automatically rotate log files as a standard utility, configured to run once per day by default.

Because event log files can grow quickly for larger Sensu installations, we recommend using logrotate to automatically rotate log files more frequently. To use the example log rotation configurations listed below, you may need to configure logrotate to run once per hour.

Log rotation for systemd

In this example, the postrotate script will reload the backend after log rotate is complete.

/var/log/sensu/events.log
{
  rotate 3
  hourly
  missingok
  notifempty
  compress
  postrotate
    /bin/systemctl reload sensu-backend.service > /dev/null 2>/dev/null || true
  endscript
}

Without the postrotate script, the backend will not reload. This will cause sensu-backend (and sensu-agent, if translated for the Sensu agent) to no longer write to the log file, even if logrotate recreates the log file.

Log rotation for sysvinit

/var/log/sensu/events.log
{
  rotate 3
  hourly
  missingok
  notifempty
  compress
  postrotate
    kill -HUP `cat /var/run/sensu/sensu-backend.pid 2> /dev/null` 2> /dev/null || true
  endscript
}

Platform metrics logging

Sensu automatically writes core platform metrics in InfluxDB Line Protocol to a file at /var/log/sensu/backend-stats.log. You can use this file as an input source for your favorite data lake solution.

Metrics logging is enabled by default but can be disabled with the disable-platform-metrics configuration option. Sensu appends updated metrics at the interval you specify with the platform-metrics-logging-interval configuration option (default is every 60 seconds).

To rotate the platform metrics log, use the same methods as for event log rotation.

Use these backend configuration options to customize platform metrics logging:

sensu-backend start --disable-platform-metrics false

disable-platform-metrics: false

sensu-backend start --platform-metrics-log-file /var/log/sensu/sensu-backend/stats.log

platform-metrics-log-file: "/var/log/sensu/sensu-backend/stats.log"

sensu-backend start --platform-metrics-logging-interval 60s

platform-metrics-logging-interval: 60s

Service management

Start the service

Use the sensu-backend tool to start the backend and apply configuration flags.

Start the backend with configuration flags:

sensu-backend start --state-dir /var/lib/sensu/sensu-backend --log-level debug

View available configuration flags and defaults:

sensu-backend start --help

If you do not include any configuration flags with the sensu-backend start command, the backend loads configuration from /etc/sensu/backend.yml by default.

Start the backend using a service manager:

sudo systemctl start sensu-backend

Stop the service

Stop the backend service using a service manager:

sudo systemctl stop sensu-backend

Restart the service

Restart the backend using a service manager:

sudo systemctl restart sensu-backend

You must restart the backend to implement any configuration updates.

Enable on boot

Enable the backend to start on system boot:

sudo systemctl enable sensu-backend

Disable the backend from starting on system boot:

sudo systemctl disable sensu-backend

Get service status

View the status of the backend service using a service manager:

sudo systemctl status sensu-backend

Get service version

Get the current backend version using the sensu-backend tool:

sensu-backend version

Get help

The sensu-backend tool provides general and command-specific help flags.

View sensu-backend commands:

sensu-backend help

List options for a specific command (in this case, sensu-backend start):

sensu-backend start --help