Events
or click any element in the pipeline to jump to it.
Events are generic containers that Sensu uses to provide context to status and metrics check results. The context, called observation data, is information about the originating entity and the corresponding status or metric check result.
These generic containers allow Sensu to handle different types of events in the pipeline for comprehensive system and service monitoring and observability. Events can contain CPU, memory, and disk usage data; custom application metrics; log data you can send to an external database; and more.
Events require a timestamp, entity, and check. Each event must contain a check result, whether status or metrics. In certain cases, an event can contain both. Because events are polymorphic in nature, it is important to never assume their content (or lack of content).
Here’s an example event that includes both status and metrics data, retrieved with sensuctl event info:
---
type: Event
api_version: core/v2
metadata:
namespace: default
spec:
check:
check_hooks: null
command: http-check --url http://localhost && http-perf --url http://localhost
--warning 1s --critical 2s
duration: 0.022274319
env_vars: null
executed: 1635959379
handlers:
- debug
high_flap_threshold: 0
history:
- executed: 1635952820
status: 0
- executed: 1635952835
status: 0
- executed: 1635952850
status: 0
- executed: 1635952865
status: 0
- executed: 1635952880
status: 0
interval: 5
is_silenced: false
issued: 1635952880
last_ok: 1635952880
low_flap_threshold: 0
metadata:
name: collect-metrics
namespace: default
occurrences: 5
occurrences_watermark: 5
output: |
http-check OK: HTTP Status 200 for http://localhost
http-perf OK: 0.001150s | dns_duration=0.000257, tls_handshake_duration=0.000000, connect_duration=0.000088, first_byte_duration=0.001131, total_request_duration=0.001150
output_metric_format: nagios_perfdata
output_metric_handlers: null
pipelines: []
processed_by: sensu-centos
proxy_entity_name: ""
publish: true
round_robin: false
runtime_assets:
- http-checks
scheduler: memory
secrets: null
state: passing
status: 0
stdin: false
subdue: null
subscriptions:
- webserver
timeout: 0
total_state_change: 0
ttl: 0
entity:
deregister: false
deregistration: {}
entity_class: agent
last_seen: 1635959379
metadata:
created_by: admin
name: sensu-centos
namespace: default
redact:
- password
- passwd
- pass
- api_key
- api_token
- access_key
- secret_key
- private_key
- secret
sensu_agent_version: 6.5.4
subscriptions:
- system
- entity:sensu-centos
- webserver
system:
arch: amd64
cloud_provider: ""
hostname: sensu-centos
libc_type: glibc
network:
interfaces:
- addresses:
- 127.0.0.1/8
- ::1/128
name: lo
- addresses:
- 10.0.2.15/24
- fe80::20b8:8cea:fa4:2e57/64
mac: 08:00:27:8b:c9:3f
name: eth0
- addresses:
- 192.168.200.95/24
- fe80::a00:27ff:fe40:ab31/64
mac: 08:00:27:40:ab:31
name: eth1
os: linux
platform: centos
platform_family: rhel
platform_version: 7.9.2009
processes: null
vm_role: guest
vm_system: vbox
user: agent
id: 12545deb-0e0f-480f-addf-34545d5a01c6
pipelines: null
sequence: 5
timestamp: 1635952880{
"type": "Event",
"api_version": "core/v2",
"metadata": {
"namespace": "default"
},
"spec": {
"check": {
"check_hooks": null,
"command": "http-check --url http://localhost && http-perf --url http://localhost --warning 1s --critical 2s",
"duration": 0.022274319,
"env_vars": null,
"executed": 1635959379,
"handlers": [
"debug"
],
"high_flap_threshold": 0,
"history": [
{
"executed": 1635952820,
"status": 0
},
{
"executed": 1635952835,
"status": 0
},
{
"executed": 1635952850,
"status": 0
},
{
"executed": 1635952865,
"status": 0
},
{
"executed": 1635952880,
"status": 0
}
],
"interval": 5,
"is_silenced": false,
"issued": 1635952880,
"last_ok": 1635952880,
"low_flap_threshold": 0,
"metadata": {
"name": "collect-metrics",
"namespace": "default"
},
"occurrences": 5,
"occurrences_watermark": 5,
"output": "http-check OK: HTTP Status 200 for http://localhost\nhttp-perf OK: 0.001150s | dns_duration=0.000257, tls_handshake_duration=0.000000, connect_duration=0.000088, first_byte_duration=0.001131, total_request_duration=0.001150\n",
"output_metric_format": "nagios_perfdata",
"output_metric_handlers": null,
"pipelines": [],
"processed_by": "sensu-centos",
"proxy_entity_name": "",
"publish": true,
"round_robin": false,
"runtime_assets": [
"http-checks"
],
"scheduler": "memory",
"secrets": null,
"state": "passing",
"status": 0,
"stdin": false,
"subdue": null,
"subscriptions": [
"webserver"
],
"timeout": 0,
"total_state_change": 0,
"ttl": 0
},
"entity": {
"deregister": false,
"deregistration": {},
"entity_class": "agent",
"last_seen": 1635959379,
"metadata": {
"created_by": "admin",
"name": "sensu-centos",
"namespace": "default"
},
"redact": [
"password",
"passwd",
"pass",
"api_key",
"api_token",
"access_key",
"secret_key",
"private_key",
"secret"
],
"sensu_agent_version": "6.5.4",
"subscriptions": [
"system",
"entity:sensu-centos",
"webserver"
],
"system": {
"arch": "amd64",
"cloud_provider": "",
"hostname": "sensu-centos",
"libc_type": "glibc",
"network": {
"interfaces": [
{
"addresses": [
"127.0.0.1/8",
":1/128"
],
"name": "lo"
},
{
"addresses": [
"10.0.2.15/24",
"fe80::20b8:8cea:fa4:2e57/64"
],
"mac": "08:00:27:8b:c9:3f",
"name": "eth0"
},
{
"addresses": [
"192.168.200.95/24",
"fe80::a00:27ff:fe40:ab31/64"
],
"mac": "08:00:27:40:ab:31",
"name": "eth1"
}
]
},
"os": "linux",
"platform": "centos",
"platform_family": "rhel",
"platform_version": "7.9.2009",
"processes": null,
"vm_role": "guest",
"vm_system": "vbox"
},
"user": "agent"
},
"id": "12545deb-0e0f-480f-addf-34545d5a01c6",
"pipelines": null,
"sequence": 5,
"timestamp": 1635952880
}
}NOTE: Metrics data points are not included in events retrieved with sensuctl event info — these events include check output text rather than a set of metrics points. To view metrics points data as shown in the event below, add a debug handler that prints events to a JSON file.
{
"entity": {
"entity_class": "agent",
"system": {
"hostname": "sensu-centos",
"os": "linux",
"platform": "centos",
"platform_family": "rhel",
"platform_version": "7.9.2009",
"network": {
"interfaces": [
{
"name": "lo",
"addresses": [
"127.0.0.1/8",
"::1/128"
]
},
{
"name": "eth0",
"mac": "08:00:27:8b:c9:3f",
"addresses": [
"10.0.2.15/24",
"fe80::20b8:8cea:fa4:2e57/64"
]
},
{
"name": "eth1",
"mac": "08:00:27:40:ab:31",
"addresses": [
"192.168.200.95/24",
"fe80::a00:27ff:fe40:ab31/64"
]
}
]
},
"arch": "amd64",
"libc_type": "glibc",
"vm_system": "vbox",
"vm_role": "guest",
"cloud_provider": "",
"processes": null
},
"subscriptions": [
"system",
"entity:sensu-centos",
"webserver"
],
"last_seen": 1635952880,
"deregister": false,
"deregistration": {},
"user": "agent",
"redact": [
"password",
"passwd",
"pass",
"api_key",
"api_token",
"access_key",
"secret_key",
"private_key",
"secret"
],
"metadata": {
"name": "sensu-centos",
"namespace": "default",
"created_by": "admin"
},
"sensu_agent_version": "6.5.4"
},
"check": {
"command": "http-check --url http://localhost \\u0026\\u0026 http-perf --url http://localhost --warning 1s --critical 2s",
"handlers": [
"debug"
],
"high_flap_threshold": 0,
"interval": 15,
"low_flap_threshold": 0,
"publish": true,
"runtime_assets": [
"http-checks"
],
"subscriptions": [
"webserver"
],
"proxy_entity_name": "",
"check_hooks": null,
"stdin": false,
"subdue": null,
"ttl": 0,
"timeout": 0,
"round_robin": false,
"duration": 0.018747388,
"executed": 1635952880,
"history": [
{
"status": 0,
"executed": 1635952820
},
{
"status": 0,
"executed": 1635952835
},
{
"status": 0,
"executed": 1635952850
},
{
"status": 0,
"executed": 1635952865
},
{
"status": 0,
"executed": 1635952880
}
],
"issued": 1635952880,
"output": "http-check OK: HTTP Status 200 for http://localhost\nhttp-perf OK: 0.001059s | dns_duration=0.000235, tls_handshake_duration=0.000000, connect_duration=0.000083, first_byte_duration=0.001040, total_request_duration=0.001059\n",
"state": "passing",
"status": 0,
"total_state_change": 0,
"last_ok": 1635952880,
"occurrences": 5,
"occurrences_watermark": 5,
"output_metric_format": "nagios_perfdata",
"output_metric_handlers": null,
"env_vars": null,
"metadata": {
"name": "collect-metrics",
"namespace": "default"
},
"secrets": null,
"is_silenced": false,
"scheduler": "memory",
"processed_by": "sensu-centos",
"pipelines": []
},
"metrics": {
"handlers": null,
"points": [
{
"name": "dns_duration",
"value": 0.000235,
"timestamp": 1635952880,
"tags": null
},
{
"name": "tls_handshake_duration",
"value": 0,
"timestamp": 1635952880,
"tags": null
},
{
"name": "connect_duration",
"value": 0.000083,
"timestamp": 1635952880,
"tags": null
},
{
"name": "first_byte_duration",
"value": 0.00104,
"timestamp": 1635952880,
"tags": null
},
{
"name": "total_request_duration",
"value": 0.001059,
"timestamp": 1635952880,
"tags": null
}
]
},
"metadata": {
"namespace": "default"
},
"id": "7cde3e3f-beee-408f-b89a-1edccd0d3edb",
"sequence": 5,
"pipelines": null,
"timestamp": 1635952880
}Checks
Checks work with the Sensu agent to produce events automatically. You can use checks to monitor server resources, services, and application health as well as collect and analyze metrics. Checks define how Sensu will process events, as well as when and where events are generated via subscriptions and scheduling.
Read Monitor server resources to learn more about using checks to generate events.
Status-only events
A Sensu event is created every time a check result is processed by the Sensu server, regardless of the status the result indicates. The agent creates an event upon receipt of the check execution result and executes any configured hooks the check might have. From there, the status result is forwarded to the Sensu backend, where it is filtered, transformed, and processed. Potentially noteworthy events may be processed by one or more event handlers, for example to send an email or invoke an automated action.
Metrics-only events
Sensu events can be created when the agent receives metrics through the StatsD listener. The agent will translate the StatsD metrics to Sensu metric format and place them inside an event. Because these events do not contain checks, they bypass the store and are sent to the event pipeline and corresponding event handlers.
Status and metrics events
Events that contain both a check and metrics most likely originated from check output metric extraction.
If a check is configured for metric extraction, the agent will parse the check output and transform it to Sensu metric format.
Both the check results and resulting (extracted) metrics are stored inside the event.
Event handlers from event.Check.Handlers and event.Metrics.Handlers will be invoked.
Proxy entities and events
You can create events with proxy entities, which are dynamically created entities that Sensu adds to the entity store if an entity does not already exist for a check result. Proxy entities allow Sensu to monitor external resources on systems where you cannot install a Sensu agent, like a network switch or website. Read Monitor external resources to learn how to use a proxy entity to monitor a website.
core/v2/events API endpoints
Sensu’s core/v2/events API endpoints provide HTTP access to create, retrieve, update, and delete events. If you create a new event that references an entity that does not already exist, the Sensu backend will automatically create a proxy entity when the event is published.
