Checks

Reference documentation

• What is a Sensu check?
  • Sensu check specification
• Check commands
  • What is a check command?
  • Check command arguments
  • How and where are check commands executed?
• Check execution platform
  • How are checks scheduled?
    • Subscription checks
    • Standalone checks
• Check results
  • What is a check result?
  • Example check result output
• Check token substitution
  • What is check token substitution?
  • Example check tokens
  • Check token specification
• Check hooks
  • What are check hooks?
  • Example check hooks
• Managing checks
• Check configuration
  • Example check definition
  • Check definition specification
    • Check naming
    • CHECK attributes
    • subdue attributes
    • influxdb attributes
    • opsgenie attributes
    • hooks attributes
    • Custom attributes
  • Check result specification
    • check attributes
    • client attributes

What is a Sensu check?

Sensu checks are commands executed by the Sensu client which monitor a condition (e.g. is Nginx running?) or collect measurements (e.g. how much disk space do I have left?). Although the Sensu client will attempt to execute any command defined for a check, successful processing of check results requires adherence to a simple specification.

Sensu check specification

• Result data is output to STDOUT or STDERR
  • For standard checks this output is typically a human-readable message
  • For metrics checks this output contains the measurements gathered by the check
• Exit status code indicates state
  • 0 indicates “OK”
  • 1 indicates “WARNING”
  • 2 indicates “CRITICAL”
  • exit status codes other than 0, 1, or 2 indicate an “UNKNOWN” or custom status

PRO TIP: Those familiar with the Nagios monitoring system may recognize this specification, as it is the same one used by Nagios plugins. As a result, Nagios plugins can be used with Sensu without any modification.

At every execution of a check command – regardless of success or failure – the Sensu client publishes the check’s result for eventual handling by the event processor (i.e. the Sensu server.

Check commands

What is a check command?

Each Sensu check definition defines a command and the interval at which it should be executed. Check commands are literally executable commands which will be executed on the Sensu client, run as the sensu user. Most check commands are provided by Sensu check plugins.

Check command arguments

Sensu check command attributes may include command line arguments for controlling the behavior of the command executable. Most Sensu check plugins provide support for command line arguments for reusability.

How and where are check commands executed?

As mentioned above, all check commands are executed by Sensu clients as the sensu user. Commands must be executable files that are discoverable on the Sensu client system (i.e. installed in a system $PATH directory).

NOTE: By default, the Sensu installer packages will modify the system $PATH for the Sensu processes to include /etc/sensu/plugins. As a result, executable scripts (e.g. plugins) located in /etc/sensu/plugins will be valid commands. This allows command attributes to use “relative paths” for Sensu plugin commands;

e.g.: "command": "check-http.rb -u https://sensuapp.org"

Check execution platform

How are checks scheduled?

Sensu offers two distinct check execution schedulers: the Sensu server, and the Sensu client (monitoring agent). The Sensu server schedules and publishes check execution requests to client subscriptions via a Publish/Subscribe model (i.e. subscription checks). The Sensu client (monitoring agent) schedules and executes standalone checks (on the local system only). Because Sensu’s execution schedulers are not mutually exclusive, any Sensu client may be configured to both schedule and execute it’s own standalone checks as well as execute subscription checks scheduled by the Sensu server.

Subscription checks

Sensu checks which are centrally defined and scheduled by the Sensu server are called “subscription checks”. Sensu’s use of a message bus (transport) for communication enables topic-based communication — where messages are published to a specific “topic”, and consumers subscribe to one or more specific topics. This form of communication is commonly referred to as the “publish-subscribe pattern”, or “pubsub” for short.

Subscription checks have a defined set of subscribers, a list of transport topics that check requests will be published to. Sensu clients become subscribers to these topics (i.e. subscriptions) via their individual client definition subscriptions attribute. In practice, subscriptions will typically correspond to a specific role and/or responsibility (e.g. a webserver, database, etc).

Subscriptions are a powerful primitives in the monitoring context because they allow you to effectively monitor for specific behaviors or characteristics corresponding to the function being provided by a particular system. For example, disk capacity thresholds might be more important (or at least different) on a database server as opposed to a webserver; conversely, CPU and/or memory usage thresholds might be more important on a caching system than on a file server. Subscriptions also allow you to configure check requests for an entire group or subgroup of systems rather than require a traditional 1:1 mapping.

Standalone checks

Sensu checks which are defined on a Sensu client with the check definition attribute standalone set to true are called “standalone checks”. The Sensu client provides its own scheduler for scheduling standalone checks which ensures scheduling consistency between Sensu clients with identical check definitions (assuming that system clocks are synchronized via NTP).

Standalone checks are an important complement to subscription checks because they provide a de-centralized management alternative for Sensu.

Check results

What is a check result?

A check result is a JSON document published as a message on the Sensu transport by the Sensu client upon completion of a check execution. Sensu check results include the check definition attributes (e.g. command, subscribers, interval, name, etc; including custom attributes), the client name the result was submitted from, and the output of the check.

Example check result output

{
  "check": {
    "status": 0,
    "command": "check-http.rb -u https://sensuapp.org",
    "subscribers": [
      "demo"
    ],
    "interval": 60,
    "name": "sensu-website",
    "issued": 1458934742,
    "executed": 1458934742,
    "duration": 0.637,
    "output": "CheckHttp OK: 200, 78572 bytes\n"
  },
  "client": "sensu-docs"
}

NOTE: please refer to the check result specification (below) for more information about check results.

Check token substitution

What is check token substitution?

Sensu check definitions may include attributes that you may wish to override on a client-by-client basis. For example, check commands – which may include command line arguments for controlling the behavior of the check command – may benefit from client-specific thresholds, etc. Sensu check tokens are check definition placeholders that will be replaced by the Sensu client with the corresponding client definition attribute values (including custom attributes).

NOTE: Sensu check tokens were formerly known as “check command tokens” (which limited token substitution to the check command attribute); command tokens were also sometimes referred to as “Sensu client overrides”; a reference to the fact that command tokens allowed client attributes to “override” check command arguments.

NOTE: Check tokens are processed before check execution, therefore token substitution will not apply to check data delivered via the local client socket input.

Example check tokens

The following is an example Sensu check definition using three check tokens for check command arguments. In this example, the check-disk-usage.rb command accepts -w (warning) and -c (critical) arguments to indicate the thresholds (as percentages) for creating warning or critical events. As configured, this check will create a warning event at 80% disk capacity, unless a different threshold is provided by the client definition (i.e. :::disk.warning|80:::); and a critical event will be created if disk capacity reaches 90%, unless a different threshold is provided by the client definition (i.e. :::disk.critical|90:::). This example also creates a custom check definition attribute called environment, which will default to a value of production, unless a different value is provided by the client definition (i.e. :::environment|production:::).

{
  "checks": {
    "check_disk_usage": {
      "command": "check-disk-usage.rb -w :::disk.warning|80::: -c :::disk.critical|90:::",
      "subscribers": [
        "production"
      ],
      "interval": 60,
      "environment": ":::environment|production:::"
    }
  }
}

The following example Sensu client definition would provide the necessary attributes to override the disk.warning, disk.critical, and environment tokens declared above.

{
  "client": {
    "name": "i-424242",
    "address": "10.0.2.100",
    "subscriptions": [
      "production",
      "webserver",
      "mysql"
    ],
    "disk": {
      "warning": 75,
      "critical": 85
    },
    "environment": "development"
  }
}

Check token specification

Token substitution syntax

Check tokens are invoked by wrapping references to client attributes with “triple colons” (i.e. three colon characters, i.e. :::). Nested Sensu client definition attributes can be accessed via “dot notation” (e.g. disk.warning).

• :::address::: would be replaced with the client address attribute
• :::url::: would be replaced with a custom attribute called url
• :::disk.warning::: would be replaced with a custom attribute called warning nested inside of a JSON hash called disk

Token substitution default values

Check token default values can be used as a fallback in the event that an attribute is not provided by the client definition. Check token default values are separated by a pipe character (|), and can be used to provide a “fallback value” for clients that are missing the declared token attribute.

• :::url|https://sensuapp.org::: would be replaced with a custom attribute called url. If no such attribute called url is included in the client definition, the default (or fallback) value of https://sensuapp.org will be used.

Unmatched tokens

If a token substitution default value is not provided (i.e. as a fallback value), and the Sensu client definition does not have a matching definition attribute, a check result indicating “unmatched tokens” will be published for the check execution (e.g.: "Unmatched check token(s): disk.warning").

Token data type limitations

As part of the substitution process, Sensu converts all tokens to strings. This means that tokens cannot be used for bare integer values or to access individual list items.

For example, token substitution cannot be used for specifying a check interval because the interval attribute requires an integer value. But token substitution can be used for alerting thresholds since those values are included within the command string.

Invalid use of token substitution:

The resulting interval value is a string ("60") instead of an integer (60), causing an invalid check request.

{
  "checks": {
    "check_disk_usage": {
      "command": "check-disk-usage.rb -w 80 -c 90",
      "interval": ":::interval|60:::"
    }
  }
}

Valid use of token substitution:

The resulting tokens are included within the command string, producing a valid check request.

{
  "checks": {
    "check_disk_usage": {
      "command": "check-disk-usage.rb -w :::disk.warning|80::: -c :::disk.critical|90:::",
      "interval": 60
    }
  }
}

When accessing a list using token substitution, Sensu returns the list in a fixed format. For example, for the following check definition executed on a client with the subscriptions ["sensu", "rhel", "all"]:

{
  "checks": {
    "token_test": {
      "command": "echo ':::subscriptions:::'",
      "standalone": true,
      "interval": 15
    }
  }
}

The resulting command output would be:

[\"sensu\", \"rhel\", \"all\"]

Check hooks

What are check hooks?

Check hooks are commands run by the Sensu client in response to the result of check command execution. The Sensu client will execute the appropriate configured hook command, depending on the check execution status (e.g. 1). Valid hook names include (in order of precedence): “1”-“255”, “ok”, “warning”, “critical”, “unknown”, and “non-zero”. The check hook command output, status, executed timestamp, and duration are captured and published in the check result. Check hook commands can optionally receive JSON serialized Sensu client and check definition data via STDIN.

Example check hooks

Check hooks can be used for automated data gathering for incident triage, for example, a check hook could be used to capture the process tree when a process has been determined to be not running etc.

{
  "checks": {
    "nginx_process": {
      "command": "check-process.rb -p nginx",
      "subscribers": [
        "proxy"
      ],
      "interval": 60,
      "hooks": {
        "non-zero": {"command": "ps aux"}
      }
    }
  }
}

Check hooks can also be used to add context to a check result, for example, outputting the last 100 lines of an error log.

{
  "checks": {
    "nginx_process": {
      "command": "check-process.rb -p nginx",
      "subscribers": [
        "proxy"
      ],
      "interval": 60,
      "hooks": {
        "critical": {"command": "tail -n 100 /var/log/nginx/error.log"}
      }
    }
  }
}

Managing checks

Delete a check

To delete a check from Sensu, remove the configuration file for the check, then restart the Sensu server and API. The following example deletes a check named check-sensu-website:

# Delete the configuration file
sudo rm /etc/sensu/conf.d/check-sensu-website.json
# Restart the Sensu server and API
sudo systemctl restart sensu-{server,api}

NOTE: On Ubuntu 14.04, CentOS 6, and RHEL 6, use sudo service sensu-{server,api} restart to restart the Sensu server and API.

While removing the configuration file removes the check from the registry, it doesn’t affect the history of the check or any existing check results. To remove a deleted check’s results and history from Sensu, use the Checks API DELETE endpoint. The following example deletes all check results and check history for a check named check-sensu-website:

# Delete check results and check history
curl -s -i -X DELETE http://127.0.0.1:4567/checks/check-sensu-website

Rename a check

To rename a check, modify the check name in the check configuration, then restart the Sensu server and API and use the Checks API DELETE endpoint to remove the history and check results tied to the former name.

Check configuration

Example check definition

The following is an example Sensu check definition, a JSON configuration file located at /etc/sensu/conf.d/check-sensu-website.json. This check definition uses a Sensu plugin named check-http.rb to ensure that the Sensu website is still available. The check is named sensu-website and it runs on Sensu clients with the production subscription, at an interval of 60 seconds.

{
  "checks": {
    "sensu-website": {
      "command": "check-http.rb -u https://sensuapp.org",
      "subscribers": [
        "production"
      ],
      "interval": 60,
      "contact": "ops"
    }
  }
}

Check definition specification

Check naming

Each check definition has a unique check name, used for the definition key. Every check definition is within the "checks": {} configuration scope.

• A unique string used to name/identify the check
• Cannot contain special characters or spaces
• Validated with Ruby regex /^[\w\.-]+$/.match("check-name")

CHECK attributes

The following attributes are configured within the {"checks": { "CHECK": {} } } configuration scope (where CHECK is a valid check name).

"type": "metric"

"command": "/etc/sensu/plugins/check-chef-client.rb"

"extension": "system_profile"

"standalone": true

"subscribers": ["production"]

"publish": false

"interval": 60

"cron": "0 0 * * *"

"timeout": 30

"stdin": true

"ttl": 100

"ttl_status": 2

"auto_resolve": false

"force_resolve": true

"handle": false

"handler": "pagerduty"

"handlers": ["pagerduty", "email"]

"low_flap_threshold": 20

"high_flap_threshold": 60

"source": "switch-dc-01"

"truncate_output": true

"truncate_output_length": 1024

"aggregate": "proxy_servers"

"aggregates": [ "webservers", "production" ]

"subdue": {}

"hooks": {}

"contact": "ops"

"contacts": ["ops"]

"proxy_requests": {}

"occurrences": 3

"refresh": 3600

"dependencies": [
  "check_app",
  "db-01/check_mysql"
]

"notification": "the shopping cart application is not responding to requests"

"description": "the shopping cart application is not responding to requests"

"influxdb": {}

"opsgenie": {}

subdue attributes

The following attributes are configured within the {"checks": { "CHECK": { "subdue": {} } } } configuration scope (where CHECK is a valid check name).

EXAMPLE

{
  "checks": {
    "check-printer": {
      "...": "...",
      "subdue": {
        "days": {
          "all": [
            {
              "begin": "5:00 PM",
              "end": "8:00 AM"
            }
          ]
        }
      }
    }
  }
}

Check subdue default times include sub-second (nanosecond) resolution. For example, a check with the following subdue attributes that executes at time 23:59:59.999 will be subdued:

{
  "days": {
    "wednesday": [
      {
        "begin": "22:30 UTC",
        "end": "11:55 UTC"
      }
    ]
  }
}

ATTRIBUTES

"days": {
  "all": [
    {
      "begin": "5:00 PM",
      "end": "8:00 AM"
    }
  ],
  "friday": [
    {
      "begin": "12:00 PM",
      "end": "5:00 PM"
    }
  ]
}

influxdb attributes

The following attributes are configured within the { "check": { "influxdb": {} } } configuration scope.

ENTERPRISE: This configuration is provided for using the built-in Sensu Enterprise InfluxDB integration.

EXAMPLE

{
  "check": {
    "name": "nginx_process",
    "...": "...",
    "influxdb": {
      "tags": {
        "dc": "us-central-1"
      }
    }
  }
}

ATTRIBUTES

{}

"tags": {
  "dc": "us-central-1"
}

opsgenie attributes

The following attributes are configured within the { "check": { "opsgenie": {} } } configuration scope.

ENTERPRISE: This configuration is provided for using the built-in Sensu Enterprise OpsGenie integration.

EXAMPLE

{
  "check": {
    "name": "nginx_process",
    "...": "...",
    "opsgenie": {
      "tags": ["production"]
    }
  }
}

ATTRIBUTES

"tags": ["production"]

hooks attributes

The following attributes are configured within the {"checks": { "CHECK": { "hooks": {} } } } configuration scope (where CHECK is a valid check name).

ATTRIBUTES

"command": "ps aux"

"timeout": 30

"stdin": true

Hook naming

Each check hook has a unique hook name. Valid hook names include (in order of precedence): “1”-“255”, “ok”, “warning”, “critical”, “unknown”, and “non-zero”. The check hook name is used to determine the appropriate hook command to execute, depending on the check execution status (e.g. 1).

HOOK attributes

The following attributes are configured within the {“checks”: { “CHECK”: { “hooks”: { “HOOK”: {}} } } } configuration scope (where CHECK is a valid check name and HOOK is a valid hook name).

EXAMPLE

{
  "checks": {
    "nginx_process": {
      "command": "check-process.rb -p nginx",
      "subscribers": [
        "proxy"
      ],
      "interval": 60,
      "hooks": {
        "non-zero": {
          "command": "ps aux",
          "timeout": 10
        }
      }
    }
  }
}

proxy_requests attributes

The following attributes are configured within the {"checks": { "CHECK": { "proxy_requests": {} } } } configuration scope (where CHECK is a valid check name).

EXAMPLE

Publish a check request to the configured subscribers (e.g. ["roundrobin:snmp_pollers"]) for every Sensu client in the registry that matches the configured client attributes in client_attributes on the configured interval (e.g. 60). Client tokens in the check definition (e.g. "check-snmp-if.rb -h :::address::: -i eth0") are substituted prior to publishing the check request. The check request check source is set to the client name.

{
  "checks": {
    "check_arista_eth0": {
      "command": "check-snmp-if.rb -h :::address::: -i eth0",
      "subscribers": [
        "roundrobin:snmp_pollers"
      ],
      "interval": 60,
      "proxy_requests": {
        "client_attributes": {
          "keepalives": false,
          "device_type": "router",
          "device_manufacturer": "arista"
        }
      }
    }
  }
}

ATTRIBUTES

"client_attributes": {
  "keepalive": false,
  "device_type": "router",
  "device_manufacturer": "arista",
  "subscriptions": "eval: value.include?('dc-01')"
}

"splay": true

"splay_coverage": 65

You can, as above, also use eval to perform more complicated filtering with Ruby on the available value, such as finding clients with particular subscriptions.

For a more general introduction to using this style of check, check out the guide on using proxy checks or the guide on adding a proxy client.

Custom attributes

Because Sensu configuration is simply JSON data, it is possible to define configuration attributes that are not part of the Sensu check definition specification. These custom check attributes are included in check results and event data, providing additional context about the check. Some great example use cases for custom check definition attributes are links to playbook documentation (i.e. “here’s a link to some instructions for how to fix this if it’s broken”), contact routing, and metric graph image URLs.

EXAMPLE

The following is an example Sensu check definition that a custom definition attribute, "playbook", a URL for documentation to aid in the resolution of events for the check. The playbook URL will be available in event data and thus able to be included in event notifications (e.g. email).

{
  "checks": {
    "check_mysql_replication": {
      "command": "check-mysql-replication-status.rb --user sensu --password secret",
      "subscribers": [
        "mysql"
      ],
      "interval": 30,
      "playbook": "http://docs.example.com/wiki/mysql-replication-playbook"
    }
  }
}

Check result specification

See check results (above) for more information about check results, including an example check result.

Required attributes below are the minimum for check results submitted via the client socket input. Additional attributes are automatically added by the client to build a complete check result.

check attributes

"status": 0

"command": "check-http.rb -u https://sensuapp.org"

"subscribers": ["database_servers"]

"interval": 60

"name": "sensu-website"

"issued": 1458934742

"executed": 1458934742

"duration": 0.637

"output": "CheckHttp OK: 200, 78572 bytes\n"

client attributes

"client": { "name": "i-424242" }