Datastore reference
Sensu stores the most recent event for each entity and check pair using either an etcd (default) or PostgreSQL database.
You can access observability event data with the Sensu web UI Events page, sensuctl event
commands, and core/v2/events API endpoints.
For longer retention of observability event data, integrate Sensu with a time-series database like InfluxDB or a searchable index like ElasticSearch or Splunk.
etcd and PostgreSQL version compatibility
Sensu requires at least etcd 3.3.2 and is tested against etcd 3.5. etcd version 3.4.0 is compatible with Sensu but may result in slower performance.
Sensu supports PostgreSQL 9.5 and later, including Amazon Relational Database Service (Amazon RDS) when configured with the PostgreSQL engine.
Use default event storage
By default, Sensu uses its embedded etcd database to store configuration and event data. This embedded database allows you to get started with Sensu without deploying a complete, scalable architecture. Sensu’s default embedded etcd configuration listens for unencrypted communication on ports 2379 (client requests) and 2380 (peer communication).
Sensu can be configured to disable the embedded etcd database and use one or more external etcd nodes for configuration and event storage instead.
To stand up an external etcd cluster, follow etcd’s clustering guide using the same store configuration.
Do not configure external etcd in Sensu via backend command line flags or the backend configuration file (/etc/sensu/backend.yml
).
As your deployment grows beyond the proof-of-concept stage, review Deployment architecture for Sensu for more information about deployment considerations and recommendations for a production-ready Sensu deployment.
If you need to upgrade your version of Etcd to match the version used by Sensu backends, please refer to Etcd’s documentation and upgrade procedures.
Scale event storage
COMMERCIAL FEATURE: Access enterprise-scale event storage in the packaged Sensu Go distribution. For more information, read Get started with commercial features.
Sensu supports using an external PostgreSQL instance for event storage in place of etcd. PostgreSQL can handle significantly higher volumes of Sensu events, which allows you to scale Sensu beyond etcd’s 8GB limit.
When configured with a PostgreSQL event store, Sensu connects to PostgreSQL to store and retrieve event data in place of etcd. Etcd continues to store Sensu entity and configuration data. You can access event data stored in PostgreSQL using the same Sensu web UI, API, and sensuctl processes as etcd-stored events.
Read the PostgreSQL documentation to install and configure PostgreSQL.
PostgreSQL requirements
For optimal performance, we recommend the following PostgreSQL configuration parameters and settings as a starting point for your postgresql.conf
file:
max_connections = 200
shared_buffers = 10GB
maintenance_work_mem = 1GB
vacuum_cost_delay = 10ms
vacuum_cost_limit = 10000
bgwriter_delay = 50ms
bgwriter_lru_maxpages = 1000
max_worker_processes = 8
max_parallel_maintenance_workers = 2
max_parallel_workers_per_gather = 2
max_parallel_workers = 8
synchronous_commit = off
wal_sync_method = fdatasync
wal_writer_delay = 5000ms
max_wal_size = 5GB
min_wal_size = 1GB
checkpoint_completion_target = 0.9
autovacuum_naptime = 10s
autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.025
Adjust the parameters and settings as needed based on your hardware and the performance you observe. Read the PostgreSQL parameters documentation for information about setting parameters.
Configure the PostgreSQL event store
At the time when you enable the PostgreSQL event store, event data cuts over from etcd to PostgreSQL. This results in a loss of recent event history. No restarts or Sensu backend configuration changes are required to enable the PostgreSQL event store.
When you successfully enable PostgreSQL as the Sensu Go event store, the Sensu backend log will include a message like this:
Mar 10 17:44:45 sensu-centos sensu-backend[1365]: {"component":"store-providers","level":"warning","msg":"switched event store to postgres","time":"2020-03-10T17:44:45Z"}
After you install and configure PostgreSQL, configure Sensu by creating a PostgresConfig
resource like the following example.
Review the datastore specification for more information.
---
type: PostgresConfig
api_version: store/v1
metadata:
name: my-postgres
spec:
batch_buffer: 0
batch_size: 1
batch_workers: 0
dsn: "postgresql://user:secret@host:port/dbname"
max_conn_lifetime: 5m
max_idle_conns: 2
pool_size: 20
strict: true
enable_round_robin: true
{
"type": "PostgresConfig",
"api_version": "store/v1",
"metadata": {
"name": "my-postgres"
},
"spec": {
"batch_buffer": 0,
"batch_size": 1,
"batch_workers": 0,
"dsn": "postgresql://user:secret@host:port/dbname",
"max_conn_lifetime": "5m",
"max_idle_conns": 2,
"pool_size": 20,
"strict": true,
"enable_round_robin": true
}
}
Save your PostgresConfig
resource definition to a file (in this example, postgres.yml
or postgres.json
).
Then, use sensuctl configured as the admin user to activate the PostgreSQL event store.
sensuctl create --file postgres.yml
sensuctl create --file postgres.json
To update your Sensu PostgreSQL configuration, repeat the sensuctl create
process.
You can expect PostgreSQL status updates in the Sensu backend logs at the warn
log level and PostgreSQL error messages in the Sensu backend logs at the error
log level.
Use environment variables to configure PostgreSQL
The Sensu backend uses the libpq library to make connections to PostgreSQL. libpq supports a number of environment variables that can be injected into the PostgreSQL data source name (DSN). Sensu loads these environment variables at runtime using the system’s environment variable file.
You can use the libpq environment variables to connect to PostgreSQL without exposing sensitive information, like usernames and passwords, in your PostgreSQL configuration. To do this, define the libpq environment variables as described in the backend reference. Sensu automatically looks up these environment variables, so you do not need to reference them in your PostgresConfig definition.
For example, to use libpq environment variables to define the PostgreSQL username, password, port, and database name in the Sensu backend environment file:
PGUSER=<PostgreSQL_username>
PGPASSWORD=<PostgreSQL_password>
PGPORT=5432
PGDATABASE=sensu_events
With these environment variables defined, the PostgreSQL configuration does not need to include the username, password, port, or database name:
---
type: PostgresConfig
api_version: store/v1
metadata:
name: postgres_datastore
spec:
dsn: "postgresql://postgres.example.com"
pool_size: 20
strict: true
{
"type": "PostgresConfig",
"api_version": "store/v1",
"metadata": {
"name": "postgres_datastore"
},
"spec": {
"dsn": "postgresql://postgres.example.com",
"pool_size": 20,
"strict": true
}
}
Disable the PostgreSQL event store
To disable the PostgreSQL event store, use sensuctl delete
with your PostgresConfig
resource definition file:
sensuctl delete --file postgres.yml
sensuctl delete --file postgres.json
The Sensu backend log will include a message to record that you successfully disabled PostgreSQL as the Sensu Go event store:
Mar 10 17:35:04 sensu-centos sensu-backend[1365]: {"component":"store-providers","level":"warning","msg":"switched event store to etcd","time":"2020-03-10T17:35:04Z"}
When you disable the PostgreSQL event store, event data cuts over from PostgreSQL to etcd, which results in a loss of recent event history. No restarts or Sensu backend configuration changes are required to disable the PostgreSQL event store.
Upgrading PostgreSQL
When upgrading PostgreSQL, we recommend following PostgreSQL’s upgrade instructions for your particular version of PostgreSQL.
Datastore specification
Top-level attributes
api_version | |
---|---|
description | Top-level attribute that specifies the Sensu API group and version. For PostgreSQL datastore configs, the api_version should be store/v1 . |
required | true |
type | String |
example |
|
metadata | |
---|---|
description | Top-level scope that contains the PostgreSQL datastore name and created_by field. |
required | true |
type | Map of key-value pairs |
example |
|
spec | |
---|---|
description | Top-level map that includes the PostgreSQL datastore config spec attributes. |
required | true |
type | Map of key-value pairs |
example |
|
type | |
---|---|
description | Top-level attribute that specifies the sensuctl create resource type. PostgreSQL datastore configs should always be type PostgresConfig . |
required | true |
type | String |
example |
|
Metadata attributes
created_by | |
---|---|
description | Username of the Sensu user who created the datastore or last updated the datastore. Sensu automatically populates the created_by field when the datastore is created or updated. |
required | false |
type | String |
example |
|
name | |
---|---|
description | PostgreSQL datastore name used internally by Sensu. |
required | true |
type | String |
example |
|
Spec attributes
batch_buffer | |
---|---|
description | Maximum number of requests to buffer in memory. WARNING: The batcher is sensitive to configuration values, and some |
required | false |
default | 0 |
type | Integer |
example |
|
batch_size | |
---|---|
description | Number of requests in each PostgreSQL write transaction, as specified in the PostgreSQL configuration. WARNING: The batcher is sensitive to configuration values, and some |
required | false |
default | 1 |
type | Integer |
example |
|
batch_workers | |
---|---|
description | Number of Goroutines sending data to PostgreSQL, as specified in the PostgreSQL configuration. WARNING: The batcher is sensitive to configuration values, and some |
required | false |
default | Current PostgreSQL pool size |
type | Integer |
example |
|
dsn | |
---|---|
description | Data source name. Specified as a URL or PostgreSQL connection string. The Sensu backend uses the Go pq library, which supports a subset of the PostgreSQL libpq connection string parameters. To avoid exposing sensitive information in the dsn attribute, configure PostgreSQL with environment variables. |
required | true |
type | String |
example |
|
enable_round_robin | |
---|---|
description | If true , enables round robin scheduling on PostgreSQL. Any existing round robin scheduling will stop and migrate to PostgreSQL as entities check in with keepalives. Sensu will gradually delete the existing etcd scheduler state as keepalives on the etcd scheduler keys expire over time. Otherwise, false .We recommend using PostgreSQL rather than etcd for round robin scheduling because etcd leases are not reliable enough to produce precise round robin behavior. |
required | false |
default | false |
type | Boolean |
example |
|
max_conn_lifetime | |
---|---|
description | Maximum time a connection can persist before being destroyed. Specify values with a numeral and a letter indicator: s to indicate seconds, m to indicate minutes, and h to indicate hours. For example, 1m , 2h , and 2h1m3s are valid. |
required | false |
type | String |
example |
|
max_idle_conns | |
---|---|
description | Maximum number of number of idle connections to retain. |
required | false |
default | 2 |
type | Integer |
example |
|
pool_size | |
---|---|
description | Maximum number of connections to hold in the PostgreSQL connection pool. We recommend 20 for most instances. |
required | false |
default | 0 (unlimited) |
type | Integer |
example |
|
strict | |
---|---|
description | If true , when the PostgresConfig resource is created, configuration validation will include connecting to the PostgreSQL database and executing a query to confirm whether the connected user has permission to create database tables. Otherwise, false .If strict: true , sensu-backend will try to connect to PostgreSQL indefinitely at 5-second intervals instead of reverting to etcd after 3 attempts.We recommend setting strict: true in most cases. If the connection fails or the user does not have permission to create database tables, resource configuration will fail and the configuration will not be persisted. This extended configuration is useful for debugging when you are not sure whether the configuration is correct or the database is working properly. |
required | false |
default | false |
type | Boolean |
example |
|