Monitoring as code with Sensu

Sensu’s end-to-end monitoring as code solution allows you to manage your monitoring and observability configurations the same way you build, test, and deploy your applications and infrastructure, like Kubernetes and Terraform. Monitoring as code combines composable building blocks with robust APIs so you can define your entire observability configuration as declarative YAML or JSON code and improve visibility, reliability, portability, and repeatability.

When a new endpoint starts up, like a cloud compute instance or Kubernetes Pod, the endpoint’s agent automatically registers it with the platform and starts collecting observability data according to the code in your configuration files. Teams can share and remix observability configurations for collecting events and metrics, diagnosing issues, sending alerts, and automatically remediating problems.

• Share, edit, review, and version your observability configuration files just like you would with other “as code” solutions, within one team or among teams across your organization.
• Maintain revision control and change history for your observability configurations.
• Export the Sensu configuration for one environment and replicate the same configuration in other environments.
• Remove, restore, back up, and recover Sensu instances based on your Sensu configuration files.
• Include your observability configuration in your centralized continuous integration/continuous delivery (CI/CD) pipeline to keep your configuration files aligned with your product and services.

To get started with monitoring as code, you’ll need a repository and configuration files that contain your resource definitions.

Create a monitoring as code repository

Create a monitoring as code repository for the configuration files that contain the Sensu resource definitions you use for monitoring and observability. You can use any source control repository.

The way you will use your configuration files will help you choose the best structure for your monitoring as code repository. For example, if you are likely to share observability components or manage your configuration files as part of your CI/CD workflow, it probably makes sense to use individual configuration files for different types of resources: one file for all of your checks, one file for all of your handlers, and so on. If you want to facilitate more granular sharing, you can save one resource definition per file.

If you want to share complete end-to-end observability configurations with your colleagues, you might save all of the resource definitions for each observability configuration in a single configuration file. This allows others to read through an entire configuration without interruption, and it’s convenient for demonstrating a complete Sensu configuration. However, a single configuration file that includes every resource type isn’t the best structure for CI/CD management or sharing resources among teams.

SensuFlow, our GitHub Action for managing Sensu resources via repository commits, requires a repository structure organized by clusters and namespaces. All resources of each type for each namespace are saved in a single configuration file:

.sensu/
  cluster/
    namespaces.yml
  namespaces/
    <namespace>/
      checks/
      hooks/
      filters/
      handlers/
      handlersets/
      mutators/
      pipelines/

Adopt a configuration file strategy

Configuration files contain your Sensu resource definitions. You can build configuration files as you go, adding resource definitions as you create them. You can also create your entire observability configuration first, then export some or all of your resource definitions to a file. Or, you can use a mix: export all of your existing resource definitions to configuration files and append new resources as you create them.

When you are ready to replicate your exported resource definitions, use sensuctl create.

Build as you go

To build as you go, use sensuctl commands to retrieve your Sensu resource definitions as you create them and copy the definitions into your configuration files.

For example, if you follow Monitor server resources and create a check named check_cpu, you can retrieve that check definition in YAML or JSON format with sensuctl:

sensuctl check info check_cpu --format yaml

sensuctl check info check_cpu --format wrapped-json

The sensuctl response will include the complete check_cpu resource definition in the specified format:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: check_cpu
spec:
  check_hooks: null
  command: check-cpu-usage -w 75 -c 90
  env_vars: null
  handlers: null
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  pipelines:
  - api_version: core/v2
    name: reduce_alerts
    type: Pipeline
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - check-cpu-usage
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system
  timeout: 0
  ttl: 0

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "check_cpu"
  },
  "spec": {
    "check_hooks": null,
    "command": "check-cpu-usage -w 75 -c 90",
    "env_vars": null,
    "handlers": null,
    "high_flap_threshold": 0,
    "interval": 60,
    "low_flap_threshold": 0,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "pipelines": [
      {
        "api_version": "core/v2",
        "name": "reduce_alerts",
        "type": "Pipeline"
      }
    ],
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "check-cpu-usage"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "system"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

If you prefer, you can also view JSON resource definitions in the Sensu web UI.

You can copy these resource definitions and paste them into manually created configuration files located anywhere on your system.

Alternatively, you can view resource definitions and copy them into a new or existing configuration file with a single sensuctl command. To use the following examples, replace <resource> with the resource type (like check) and replace <resource_name> with the name of the resource (like check_cpu).

• Copy the resource defintion to a new file (or overwrite an existing file with the same name):

  sensuctl <resource> info <resource_name> --format yaml > resource.yml

  sensuctl <resource> info <resource_name> --format wrapped-json > resource.json

• Copy the resource defintion to a new file (or overwrite an existing file with the same name) and show the resource definition in stdout:

  sensuctl <resource> info <resource_name> --format yaml | tee resource.yml

  sensuctl <resource> info <resource_name> --format wrapped-json | tee resource.json

• Append the resource defintion to an existing file:

  sensuctl <resource> info <resource_name> --format yaml >> resource.yml

  sensuctl <resource> info <resource_name> --format wrapped-json >> resource.json

• Append the resource defintion to an existing file and show the resource definition in stdout:

  sensuctl <resource> info <resource_name> --format yaml | tee -a resource.yml

  sensuctl <resource> info <resource_name> --format wrapped-json | tee -a resource.json

Export existing resources

If you’ve already created observability resources, use sensuctl dump to create a copy of your existing resource definitions.

First, create a sensu directory:

mkdir sensu

Then, copy your observability resource definitions according to the repository structure you are using. For example, if you want to save resources according to type and namespace, this command will save all of your check definitions for the production namespace in one configuration file:

sensuctl dump core/v2.CheckConfig \
--namespace production \
--format yaml | > sensu/namespaces/production/checks.yml

sensuctl dump core/v2.CheckConfig \
--namespace production \
--format wrapped-json | > sensu/namespaces/production/checks.json

Repeat this command for each resource type in each of your namespaces.

Strip namespaces from resource definitions

To replicate and reuse resources in any namespace without manual editing, create a copy of your existing resources with the namespaces stripped from their definitions:

sensuctl dump all \
--all-namespaces \
--format yaml | grep -v "^\s*namespace:" > sensu/resources.yml

sensuctl dump all \
--all-namespaces \
--format wrapped-json | grep -v "^\s*namespace:" > sensu/resources.json

Best practices for monitoring as code

Sensu’s monitoring as code solution is flexible — you can use any source control repository and choose your own directory structure — but following a few best practices will contribute to a successful implementation.

• To maintain consistency, save all of your resources as only one file type: YAML or JSON.
• Include all dependencies within a resource definition. For example, if a handler requires a dynamic runtime asset and a secret, include the asset and secret definitions with the definition for the handler itself.
• Choose the labels you use in your resource definitions with care. CI/CD systems like SensuFlow rely on labels to determine which resources to delete, so if all of your resources have the same labels, you could delete resources you didn’t intend to be managed in a particular CI/CD workflow.
• Establish a resource-labeling schema throughout your organization to facilitate CI/CD. Following the same method for applying labels helps keep unmanaged Sensu resources from multiplying and allows different teams to confidently deploy their own CI/CD workflows without the risk of accidentally deleting another team’s resources.

Implement CI/CD with monitoring as code

When you’re ready, expand your monitoring as code practices to include managing your Sensu configuration files with a CI/CD workflow. CI/CD takes the manual work out of maintaining and updating your monitoring as code repository so that any updates to the Sensu resources in your monitoring as code repository are reflected in your Sensu configuration in a timely manner.

If you’re already using CI/CD, you already have workflows for versioning, building, testing, and deploying your code. Integrating monitoring as code means your monitoring and observability can go through those same CI/CD workflows.

There’s no one “correct” way to implement CI/CD with monitoring as code, but the SensuFlow GitHub Action offers a turnkey reference implementation that helps you create your own monitoring as code workflow and start managing Sensu resources via repository commits.

Use SensuFlow for CI/CD monitoring as code

SensuFlow is a git-based, prescriptive monitoring as code workflow that uses sensuctl (including sensuctl prune) to synchronize your monitoring and observability code with your Sensu deployments.

SensuFlow requires:

• A code repository of Sensu resource definitions
• A Sensu role-based access control (RBAC) service account with permission to manage all resources in your repository
• A resource labeling convention to designate which resources the SensuFlow workflow should manage
• Integration with your CI/CD system to run sensuctl commands as the service account user from the repository of resource definitions

Read the SensuFlow GitHub Action marketplace page and Monitoring as code with Sensu Go and SensuFlow to get started with SensuFlow as your monitoring as code workflow.