Send data to Sumo Logic with Sensu

Follow this guide to create a pipeline that sends data from a Sensu check to Sumo Logic for long-term logs and metrics storage. Sensu checks are commands the Sensu agent executes that generate observability data in a status or metric event. Sensu pipelines define the event filters and actions the Sensu backend executes on the events.

Requirements

To follow this guide, install the Sensu backend, make sure at least one Sensu agent is running, and configure sensuctl to connect to the backend as the admin user.

The examples in this guide rely on the check_cpu check from Monitor server resources with checks. Before you begin, follow the instructions to add the sensu/check-cpu-usage dynamic runtime asset and the check_cpu check.

Configure a Sensu entity

Sensu checks have a subscriptions attribute, where you specify strings to indicate which subscribers will execute the checks. For Sensu to execute a check, at least one entity must include a subscription that matches a subscription in the check definition. In the example in this guide, the check_cpu check includes the system subscription, so at least one entity must subscribe to system to run the check.

First, select the entity whose data you want to send to Sumo Logic. To list all of your entities in the current namespace, run:

sensuctl entity list

The ID in the response is the entity name. Select one of the listed entities.

Before you run the following sensuctl command, replace <ENTITY_NAME> with the name of your entity. Then run the command to add the system subscription to your entity:

sensuctl entity update <ENTITY_NAME>

• For Entity Class, press enter.
• For Subscriptions, type system and press enter.

Finally, confirm that both Sensu services are running:

systemctl status sensu-backend && systemctl status sensu-agent

The response should indicate active (running) for both the Sensu backend and agent.

Register the dynamic runtime asset

The sensu/sensu-sumologic-handler dynamic runtime asset includes the scripts your handler will need to send observability data to Sumo Logic.

To add the sensu/sensu-sumologic-handler asset, run:

sensuctl asset add sensu/sensu-sumologic-handler:0.3.0 -r sumologic-handler

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: sumologic-handler.

The response will indicate that the asset was added:

fetching bonsai asset: sensu/sensu-sumologic-handler:0.3.0
added asset: sensu/sensu-sumologic-handler:0.3.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sumologic-handler"].

To confirm that the asset was added to your Sensu backend, run:

sensuctl asset info sumologic-handler

The response will list the available builds for the sensu/sensu-sumologic-handler dynamic runtime asset.

Set up an HTTP Logs and Metrics Source

Set up a Sumo Logic HTTP Logs and Metrics Source to collect your Sensu observability data.

Log in to your Sumo Logic account and follow these instructions:

1. In the Sumo Logic left-navigation menu, click Manage Data and then Collection to open the Collection tab.

   

2. At the top-right of the Collection tab, click Add Collector.

   

3. In the Click Selector Type modal window, click Hosted Collector.

   

4. In the Add Hosted Collector modal window:

   • Type sensu in the Name field.
   • Click Save.

   

5. In the Confirm prompt, click OK.

   

6. Under Cloud APIs, click HTTP Logs & Metrics.

   

7. In the HTTP Logs & Metrics form:

   • Type sensu-http in the Name field.
   • Type sensu-events in the Source Category field.
   • Click Save.

   

8. In the HTTP Source Address prompt, copy the listed URL and click OK. You will use this URL in the next step as the SUMOLOGIC_URL value for the secret in your Sensu handler definition.

   

Add the Sumo Logic handler

Now that you’ve set up a Sumo Logic HTTP Logs and Metrics Source, you can create a handler that uses the sensu/sensu-sumologic-handler dynamic runtime asset to send observability data to Sumo Logic.

The Sensu Sumo Logic Handler asset requires a SUMOLOGIC_URL variable. The value for the SUMOLOGIC_URL variable is the Sumo Logic HTTP Source Address URL, which you retrieved in the last step of setting up an HTTP Logs and Metrics Source.

Configure the SUMOLOGIC_URL environment variable

To save your Sumo Logic HTTP Source Address URL as an environment variable:

1. Create the files from which the sensu-backend service will read environment variables. If you have already created this file on your system, skip to step 2.

   sudo touch /etc/default/sensu-backend

   sudo touch /etc/sysconfig/sensu-backend

2. In the following code, replace <SumoLogic_HTTPSourceAddress_URL> with your Sumo Logic HTTP Source Address URL. Run:

   echo 'SUMOLOGIC_URL=<SumoLogic_HTTPSourceAddress_URL>' | sudo tee -a /etc/default/sensu-backend

   echo 'SUMOLOGIC_URL=<SumoLogic_HTTPSourceAddress_URL>' | sudo tee -a /etc/sysconfig/sensu-backend

3. Restart the sensu-backend:

   sudo systemctl restart sensu-backend

This configures the SUMOLOGIC_URL environment variable to your Sumo Logic HTTP Source Address URL in the context of the sensu-backend process.

Create the Env secret

Create a secret named sumologic_url that refers to the environment variable ID SUMOLOGIC_URL. Run:

cat << EOF | sensuctl create
---
type: Secret
api_version: secrets/v1
metadata:
  name: sumologic_url
spec:
  id: SUMOLOGIC_URL
  provider: env
EOF

cat << EOF | sensuctl create
{
  "type": "Secret",
  "api_version": "secrets/v1",
  "metadata": {
    "name": "sumologic_url"
  },
  "spec": {
    "id": "SUMOLOGIC_URL",
    "provider": "env"
  }
}
EOF

Now you can refer to the sumologic_url secret in your handler to securely pass your Sumo Logic HTTP Source Address URL.

Create a Sumo Logic handler

Run the following command to create a handler to send Sensu observability data to your Sumo Logic HTTP Logs and Metrics Source:

cat << EOF | sensuctl create
---
type: Handler
api_version: core/v2
metadata:
  name: sumologic
spec:
  command: >-
    sensu-sumologic-handler --send-log --send-metrics
    --source-host "{{ .Entity.Name }}"
    --source-name "{{ .Check.Name }}"
  type: pipe
  runtime_assets:
  - sumologic-handler
  secrets:
  - name: SUMOLOGIC_URL
    secret: sumologic_url
EOF

cat << EOF | sensuctl create
{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "sumologic"
  },
  "spec": {
    "command": "sensu-sumologic-handler --send-log --send-metrics --source-host \"{{ .Entity.Name }}\" --source-name \"{{ .Check.Name }}\"",
    "type": "pipe",
    "runtime_assets": [
      "sumologic-handler"
    ],
    "secrets": [
      {
        "name": "SUMOLOGIC_URL",
        "secret": "sumologic_url"
      }
    ]
  }
}
EOF

Make sure that your handler was added by retrieving the complete handler definition in YAML or JSON format:

sensuctl handler info sumologic --format yaml

sensuctl handler info sumologic --format wrapped-json

The response will list the complete handler resource definition:

---
type: Handler
api_version: core/v2
metadata:
  name: sumologic
spec:
  command: sensu-sumologic-handler --send-log --send-metrics --source-host "{{ .Entity.Name }}" --source-name "{{ .Check.Name }}"
  env_vars: null
  filters: null
  handlers: null
  runtime_assets:
  - sumologic-handler
  secrets:
  - name: SUMOLOGIC_URL
    secret: sumologic_url
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "sumologic"
  },
  "spec": {
    "command": "sensu-sumologic-handler --send-log --send-metrics --source-host \"{{ .Entity.Name }}\" --source-name \"{{ .Check.Name }}\"",
    "env_vars": null,
    "filters": null,
    "handlers": null,
    "runtime_assets": [
      "sumologic-handler"
    ],
    "secrets": [
      {
        "name": "SUMOLOGIC_URL",
        "secret": "sumologic_url"
      }
    ],
    "timeout": 0,
    "type": "pipe"
  }
}

Create a pipeline with the Sumo Logic handler

With your Sumo Logic handler configured, you can add it to a pipeline workflow. A single pipeline workflow can include one or more event filters, one mutator, and one handler.

To send data for all events (as opposed to only incidents), create a pipeline that includes only the Sumo Logic handler you’ve already configured and the built-in not_silenced event filter — no mutators. To add the pipeline, run:

cat << EOF | sensuctl create
---
type: Pipeline
api_version: core/v2
metadata:
  name: sensu_to_sumo
spec:
  workflows:
  - name: logs_to_sumologic
    filters:
    - name: not_silenced
      type: EventFilter
      api_version: core/v2
    handler:
      name: sumologic
      type: Handler
      api_version: core/v2
EOF

cat << EOF | sensuctl create
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "sensu_to_sumo"
  },
  "spec": {
    "workflows": [
      {
        "name": "logs_to_sumologic",
        "filters": [
          {
            "name": "not_silenced",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "sumologic",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}
EOF

Assign the pipeline to a check

To use the sensu_to_sumo pipeline, list it in a check definition’s pipelines array. This example uses the check_cpu check created in Monitor server resources, but you can add the pipeline to any Sensu check you wish. All the observability events that the check produces will be processed according to the pipeline’s workflows.

Assign your sensu_to_sumo pipeline to the check_cpu check to start sending Sensu data to Sumo Logic.

To open the check definition in your text editor, run:

sensuctl edit check check_cpu

Replace the pipelines: [] line with the following array:

  pipelines:
  - type: Pipeline
    api_version: core/v2
    name: sensu_to_sumo

To confirm that the updated check_cpu resource definition includes the pipelines reference, run:

sensuctl check info check_cpu --format yaml

sensuctl check info check_cpu --format wrapped-json

The updated check definition will be similar to this example:

---
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  name: check_cpu
spec:
  check_hooks: null
  command: check-cpu-usage -w 75 -c 90
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 15
  low_flap_threshold: 0
  output_metric_format: prometheus_text
  output_metric_handlers: null
  pipelines:
  - api_version: core/v2
    name: sensu_to_sumo
    type: Pipeline
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - check-cpu-usage
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system
  timeout: 0
  ttl: 0

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "check_cpu",
    "created_by": "admin"
  },
  "spec": {
    "check_hooks": null,
    "command": "check-cpu-usage -w 75 -c 90",
    "env_vars": null,
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 15,
    "low_flap_threshold": 0,
    "output_metric_format": "prometheus_text",
    "output_metric_handlers": null,
    "pipelines": [
      {
        "api_version": "core/v2",
        "name": "sensu_to_sumo",
        "type": "Pipeline"
      }
    ],
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "check-cpu-usage"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "system"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

View your Sensu data in Sumo Logic

It will take a few moments after you add the pipeline to the check for your Sensu observability data to appear in Sumo Logic. Use the Live Tail feature to confirm that your data is reaching Sumo Logic.

1. In Sumo Logic, click the + New button and select Live Tail from the drop-down menu.

   

2. In the Live Tail search field, enter _collector=sensu and click Run.

   

Within a few seconds, the Live Tail page should begin to display your Sensu observability data.

If you see Sensu data on the Live Tail page, well done! You have a successful workflow that sends Sensu observability data to your Sumo Logic account.

What’s next

To share and reuse the check, handler, and pipeline like code, save them to files and start building a monitoring as code repository.

Learn more about the sensu/sensu-sumologic-handler dynamic runtime asset. You can also configure a Sumo Logic dashboard to search, view, and analyze the Sensu data you’re sending to your Sumo Logic HTTP Logs and Metrics Source.

In addition to the traditional handler we used in this example, you can use Sensu Plus, our built-in integration, to send metrics to Sumo Logic with a streaming Sumo Logic metrics handler.