TCP stream handlers reference

COMMERCIAL FEATURE: Access TCP stream handlers in the packaged Sensu Go distribution. For more information, read Get started with commercial features.

Sensu executes TCP stream handlers during the process stage of the observability pipeline.

Like traditional TCP handlers, TCP stream handlers send observability event data to TCP sockets for external services to consume. However, TCP stream handlers can help prevent the data bottlenecks you may experience with traditional TCP handlers.

Traditional TCP handlers start a new UNIX process for every Sensu event they receive and require a new connection to send every event. As you scale up and process more events per second, the rate at which the TCP handler can transmit observability event data decreases.

TCP stream handlers allow you to configure a connection pool with a maximum number of connections for the handler to use. For example, suppose you configure a TCP stream handler with a pool of 10 connections, and 1000 events are queued for transmission. As each connection finishes transmitting an event, it becomes available again and returns to the pool so the handler can use it to send the next event in the queue.

TCP stream handlers will reuse the available connections as long as they can rather than requiring a new connection for every event, which increases event throughput. In addition to providing a persistent TCP connection to transmit Sensu observation events to a remote data storage service, TCP stream handlers allow you to use transport layer security (TLS) for secure data transmission.

TCP stream handlers are commercial resources available for use in pipeline definitions.

TCP stream handler example

This example shows a TCP stream handler resource definition configured to use TLS:

---
type: TCPStreamHandler
api_version: pipeline/v1
metadata:
  name: logstash
spec:
  address: 127.0.0.1:4242
  tls_ca_cert_file: "/path/to/tls/ca.pem"
  tls_cert_file: "/path/to/tls/cert.pem"
  tls_key_file: "/path/to/tls/key.pem"
  max_connections: 10
  min_reconnect_delay: 10ms
  max_reconnect_delay: 10s
{
  "type": "TCPStreamHandler",
  "api_version": "pipeline/v1",
  "metadata": {
    "name": "logstash"
  },
  "spec": {
    "address": "127.0.0.1:4242",
    "tls_ca_cert_file": "/path/to/tls/ca.pem",
    "tls_cert_file": "/path/to/tls/cert.pem",
    "tls_key_file": "/path/to/tls/key.pem",
    "max_connections": 10,
    "min_reconnect_delay": "10ms",
    "max_reconnect_delay": "10s"
  }
}

Use TCP stream handlers

TCP stream handlers are commercial resources and are available for use only in pipelines.

NOTE: TCP stream handlers are not used by listing the handler name in the check handlers attribute.

To use a TCP stream handler, list it as the handler in a pipeline definition. For example, this pipeline definition uses the logstash example along with the built-in is_incident event filter:

---
type: Pipeline
api_version: core/v2
metadata:
  name: tcp_logging_workflows
spec:
  workflows:
  - name: log_all_incidents
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    handler:
      name: logstash
      type: TCPStreamHandler
      api_version: pipeline/v1
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "tcp_logging_workflows"
  },
  "spec": {
    "workflows": [
      {
        "name": "log_all_incidents",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "logstash",
          "type": "TCPStreamHandler",
          "api_version": "pipeline/v1"
        }
      }
    ]
  }
}

TCP stream handler specification

Top-level attributes

type
description Top-level attribute that specifies the sensuctl create resource type. TCP stream handlers should always be type TCPStreamHandler.
required Required for TCP stream handler definitions in wrapped-json or yaml format for use with sensuctl create.
type String
example 
type: TCPStreamHandler
{
  "type": "TCPStreamHandler"
}
api_version
description Top-level attribute that specifies the Sensu API group and version. For TCP stream handlers in this version of Sensu, the api_version should always be pipeline/v1.
required Required for TCP stream handler definitions in wrapped-json or yaml format for use with sensuctl create.
type String
example 
api_version: pipeline/v1
{
  "api_version": "pipeline/v1"
}
metadata
description Top-level collection of metadata about the TCP stream handler that includes name, namespace, and created_by as well as custom labels and annotations. The metadata map is always at the top level of the handler definition. This means that in wrapped-json and yaml formats, the metadata scope occurs outside the spec scope. Read metadata attributes for details.
required Required for TCP stream handler definitions in wrapped-json or yaml format for use with sensuctl create.
type Map of key-value pairs
example 
metadata:
  name: logstash
  namespace: default
  created_by: admin
  labels:
    environment: development
    region: us-west-2
  annotations:
    managed-by: ops
{
  "metadata": {
    "name": "logstash",
    "namespace": "default",
    "created_by": "admin",
    "labels": {
      "environment": "development",
      "region": "us-west-2"
    },
    "annotations": {
      "managed-by": "ops"
    }
  }
}
spec
description Top-level map that includes the TCP stream handler spec attributes.
required Required for TCP stream handler definitions in wrapped-json or yaml format for use with sensuctl create.
type Map of key-value pairs
example 
spec:
  address: 127.0.0.1:4242
  tls_ca_cert_file: "/path/to/tls/ca.pem"
  tls_cert_file: "/path/to/tls/cert.pem"
  tls_key_file: "/path/to/tls/key.pem"
  max_connections: 10
  min_reconnect_delay: 10ms
  max_reconnect_delay: 10s
{
  "spec": {
    "address": "127.0.0.1:4242",
    "tls_ca_cert_file": "/path/to/tls/ca.pem",
    "tls_cert_file": "/path/to/tls/cert.pem",
    "tls_key_file": "/path/to/tls/key.pem",
    "max_connections": 10,
    "min_reconnect_delay": "10ms",
    "max_reconnect_delay": "10s"
  }
}

Metadata attributes

name
description Unique string used to identify the TCP stream handler. TCP stream handler names cannot contain special characters or spaces (validated with Go regex \A[\w\.\-]+\z). Each TCP stream handler must have a unique name within its namespace.
required true
type String
example 
name: logstash
{
  "name": "logstash"
}
namespace
description Sensu RBAC namespace that the TCP stream handler belongs to.
required false
type String
default default
example 
namespace: default
{
  "namespace": "default"
}
created_by
description Username of the Sensu user who created the TCP stream handler or last updated the TCP stream handler. Sensu automatically populates the created_by field when the TCP stream handler is created or updated.
required false
type String
example 
created_by: admin
{
  "created_by": "admin"
}
labels
description Custom attributes to include with observation event data that you can use for response and web UI view filtering.

If you include labels in your event data, you can filter API responses, sensuctl responses, and web UI views based on them. In other words, labels allow you to create meaningful groupings for your data.

Limit labels to metadata you need to use for response filtering. For complex, non-identifying metadata that you will not need to use in response filtering, use annotations rather than labels.
required false
type Map of key-value pairs. Keys can contain only letters, numbers, and underscores and must start with a letter. Values can be any valid UTF-8 string.
default null
example 
labels:
  environment: development
  region: us-west-2
{
  "labels": {
    "environment": "development",
    "region": "us-west-2"
  }
}
annotations
description Non-identifying metadata to include with observation event data that you can access with event filters. You can use annotations to add data that’s meaningful to people or external tools that interact with Sensu.

In contrast to labels, you cannot use annotations in API response filtering, sensuctl response filtering, or web UI views.
required false
type Map of key-value pairs. Keys and values can be any valid UTF-8 string.
default null
example 
annotations:
  managed-by: ops
{
  "annotations": {
    "managed-by": "ops"
  }
}

Spec attributes

address
description The hostname:port combination the TCP stream handler should connect to.
required true
type String
example 
address: 127.0.0.1:4242
{
  "address": "127.0.0.1:4242"
}
tls_ca_cert_file
description Path to the PEM-format CA certificate to use for TLS client authentication.
required false
type String
example 
tls_ca_cert_file: "/path/to/tls/ca.pem"
{
  "tls_ca_cert_file": "/path/to/tls/ca.pem"
}
tls_cert_file
description Path to the PEM-format certificate to use for TLS client authentication. This certificate and its corresponding key are required for secure client communication.
required false
type String
example 
tls_cert_file: "/path/to/tls/cert.pem"
{
  "tls_cert_file": "/path/to/tls/cert.pem"
}
tls_key_file
description Path to the PEM-format key file associated with the tls_cert_file to use for TLS client authentication. This key and its corresponding certificate are required for secure client communication.
required false
type String
example 
tls_key_file: "/path/to/tls/key.pem"
{
  "tls_key_file": "/path/to/tls/key.pem"
}
max_connections
description Maximum number of connections to keep alive in the connection pool. If set to 0, connection pooling is disabled.
required true
type Integer
example 
max_connections: 10
{
  "max_connections": 10
}
max_reconnect_delay
description Maximum time to wait while retrying a broken connection. In seconds (s) or milliseconds (ms).
required true
type String
example 
max_reconnect_delay: 10s
{
  "max_reconnect_delay": "10s"
}
min_reconnect_delay
description Minimum time to wait while retrying a broken connection. In seconds (s) or milliseconds (ms).
required true
type String
example 
min_reconnect_delay: 10ms
{
  "min_reconnect_delay": "10ms"
}