Multi-cluster visibility with federation
COMMERCIAL FEATURE: Access federation in the packaged Sensu Go distribution. For more information, read Get started with commercial features.
Sensu’s enterprise/federation/v1 API endpoints allow you to register external clusters, gain single-pane-of-glass visibility into the health of your infrastructure and services across multiple distinct Sensu instances within the web UI, and mirror your changes in one cluster to follower clusters. This is useful when you want to provide a single entry point for Sensu users who need to manage monitoring across multiple distinct physical data centers, cloud regions, or providers.
After you configure federation, you can also create, update, and delete clusters using sensuctl create, edit, and delete commands.
Federation is not enabled by default. You must create a cluster resource for the federation cluster and register it. Only cluster administrators can register a new cluster, but every user can query the list of clusters.
Complete federation of multiple Sensu instances relies on a combination of features:
Feature | Purpose in federation |
---|---|
JSON Web Token (JWT) authentication | Cross-cluster token authentication using asymmetric key encryption |
etcd replicators | Replicate RBAC policy across clusters and namespaces |
Federation Gateway and APIs | Configure federation access for cross-cluster visibility in web UI |
Follow the example in this guide to configure these features. The example assumes you wish to federate three named Sensu clusters:
Cluster name | Hostname |
---|---|
gateway |
sensu.gateway.example.com |
alpha |
sensu.alpha.example.com |
beta |
sensu.beta.example.com |
The gateway
cluster will be the entry point for operators to manage Sensu resources in the alpha
and beta
clusters.
This guide assumes a single sensu-backend in each cluster, but named clusters composed of multiple sensu-backends are supported.
This diagram depicts the federation relationship documented in this guide:
Complete the steps in this guide to browse events, entities, checks, and other resources in the gateway
, alpha
, and beta
clusters from the gateway
cluster web UI.
Requirements
Because federation depends on communication with multiple disparate clusters, working TLS is required for successful federated operation.
To ensure that cluster members can validate each other, certificates for each cluster member should include the IP addresses or hostnames specified in the values of sensu-backend etcd-advertise-client-urls
, etcd-advertise-peer-urls
, and etcd-initial-advertise-peer-urls
parameters.
In addition to the certificate’s Common Name (CN), Subject Alternative Names (SANs) are also honored for validation.
NOTE: As of Go 1.15, certificates must include their CN as an SAN field. To prevent connection errors, follow Generate certificates to make sure your certificates’ SAN fields include their CNs.
To continue with this guide, make sure you have the required TLS credentials in place:
- A PEM-formatted X.509 certificate and corresponding private key copied to each cluster member.
- A corresponding certificate authority (CA) certificate chain copied to each cluster member.
If you don’t have existing infrastructure for issuing certificates, read Generate certificates for our recommended self-signed certificate issuance process.
This prerequisite extends to configuring the following Sensu backend etcd parameters:
Backend property | Description |
---|---|
etcd-cert-file |
Path to certificate used for TLS on etcd client/peer communications (for example, /etc/sensu/tls/backend-1.example.com.pem . |
etcd-key-file |
Path to key corresponding with etcd-cert-file certificate (for example, /etc/sensu/tls/backend-1-key.example.com.pem . |
etcd-trusted-ca-file |
Path to CA certificate chain file (for example, /etc/sensu/tls/ca.pem . This CA certificate chain must be usable to validate certificates for all backends in the federation. |
etcd-client-cert-auth |
Enforces certificate validation to authenticate etcd replicator connections. Set to true to secure etcd communication. |
etcd-advertise-client-urls |
List of https URLs to advertise for etcd replicators, accessible by other backends in the federation (for example, https://sensu.beta.example.com:2379 ). |
etcd-listen-client-urls |
List of https URLs to listen on for etcd replicators (for example, https://0.0.0.0:2379 to listen on port 2379 across all ipv4 interfaces). |
WARNING: You must provide an explicit, non-default etcd configuration to secure etcd communication in transit.
If you do not properly configure secure etcd communication, your Sensu configuration will be vulnerable to unauthorized manipulation via etcd client connections.
This includes providing non-default values for the etcd-advertise-client-urls
and etcd-listen-client-urls
backend parameters and creating a certificate and key for the etcd-cert-file
and etcd-key-file
values.
The default values are not suitable for use under federation.
Configure shared token signing keys
Whether federated or standalone, Sensu backends issue JSON Web Tokens (JWTs) to users upon successful authentication. These tokens include a payload that describes the username and group affiliations. The payload is used to determine permissions based on the configured RBAC policy.
In a federation of Sensu backends, each backend needs to have the same public/private key pair. These asymmetric keys are used to crypotgraphically vouch for the user’s identity in the JWT payload. Using shared JWT keys enables clusters to grant users access to Sensu resources according to their local policies but without requiring user resources to be present uniformly across all clusters in the federation.
By default, a Sensu backend automatically generates an asymmetric key pair for signing JWTs and stores it in the etcd database. When configuring federation, you must generate keys as files on disk so they can be copied to all backends in the federation.
-
Use the
openssl
command line tool to generate a P-256 elliptic curve private key:openssl ecparam -genkey -name prime256v1 -noout -out jwt_private.pem
-
Generate a public key from the private key:
openssl ec -in jwt_private.pem -pubout -out jwt_public.pem
-
Save the JWT keys in
/etc/sensu/certs
on each cluster backend. -
Add the
jwt-private-key-file
andjwt-public-key-file
attributes in/etc/sensu/backend.yml
and specify the paths to the JWT private and public keys:jwt-private-key-file: /etc/sensu/certs/jwt_private.pem jwt-public-key-file: /etc/sensu/certs/jwt_public.pem
-
Restart the Sensu backend so that your settings take effect:
sudo systemctl restart sensu-backend
Add a user and a cluster role binding
To test your configuration, provision a user and a cluster role binding in the gateway
cluster.
-
Confirm that sensuctl is configured to communicate with the
gateway
cluster:sensuctl config view
The response will list the active configuration:
=== Active Configuration API URL: https://sensu.gateway.example.com:8080 Namespace: default Format: tabular Username: admin
-
Create a
federation-viewer
user:sensuctl user create federation-viewer --interactive
-
When prompted for username and groups, press enter.
-
When prompted for password, enter a password for the
federation-viewer
user. Make a note of the password you entered — you’ll use it to log in to the web UI after you configure RBAC policy replication and registered clusters into your federation.This creates the following user:
username: federation-viewer disabled: false
{ "username": "federation-viewer", "disabled": false }
-
Grant the
federation-viewer
user read-only access with a cluster role binding for the built-inview
cluster role:sensuctl cluster-role-binding create federation-viewer-readonly --cluster-role=view --user=federation-viewer
This command creates the following cluster role binding resource definition:
--- type: ClusterRoleBinding api_version: core/v2 metadata: created_by: admin name: federation-viewer-readonly spec: role_ref: name: view type: ClusterRole subjects: - name: federation-viewer type: User
{ "type": "ClusterRoleBinding", "api_version": "core/v2", "metadata": { "created_by": "admin", "name": "federation-viewer-readonly" }, "spec": { "role_ref": { "name": "view", "type": "ClusterRole" }, "subjects": [ { "name": "federation-viewer", "type": "User" } ] } }
Create etcd replicators
Etcd replicators use the etcd make-mirror utility for one-way replication of Sensu RBAC policy resources.
This allows you to centrally define RBAC policy on the gateway
cluster and replicate RBAC resources to other clusters in the federation (alpha
and beta
), ensuring consistent permissions for Sensu users across multiple clusters via the gateway
web UI.
-
Configure one etcd replicator per cluster for each RBAC policy resource, across all namespaces, for each backend in the federation.
NOTE: Create a replicator for each resource type you want to replicate. Replicating
namespace
resources will not replicate the Sensu resources that belong to those namespaces.
The etcd replicators reference includes examples you can follow forRole
,RoleBinding
,ClusterRole
, andClusterRoleBinding
resources.In this example, the following etcd replicator resources will replicate ClusterRoleBinding resources from the
gateway
cluster to the two target clusters:--- api_version: federation/v1 type: EtcdReplicator metadata: name: AlphaClusterRoleBindings spec: ca_cert: "/etc/sensu/certs/ca.pem" cert: "/etc/sensu/certs/gateway.pem" key: "/etc/sensu/certs/gateway-key.pem" url: https://sensu.alpha.example.com:2379 api_version: core/v2 resource: ClusterRoleBinding replication_interval_seconds: 30
{ "api_version": "federation/v1", "type": "EtcdReplicator", "metadata": { "name": "AlphaClusterRoleBindings" }, "spec": { "ca_cert": "/etc/sensu/certs/ca.pem", "cert": "/etc/sensu/certs/gateway.pem", "key": "/etc/sensu/certs/gateway-key.pem", "url": "https://sensu.alpha.example.com:2379", "api_version": "core/v2", "resource": "ClusterRoleBinding", "replication_interval_seconds": 30 } }
--- api_version: federation/v1 type: EtcdReplicator metadata: name: BetaClusterRoleBindings spec: ca_cert: "/etc/sensu/certs/ca.pem" cert: "/etc/sensu/certs/gateway.pem" key: "/etc/sensu/certs/gateway-key.pem" url: https://sensu.beta.example.com:2379 api_version: core/v2 resource: ClusterRoleBinding replication_interval_seconds: 30
{ "api_version": "federation/v1", "type": "EtcdReplicator", "metadata": { "name": "BetaClusterRoleBindings" }, "spec": { "ca_cert": "/etc/sensu/certs/ca.pem", "cert": "/etc/sensu/certs/gateway.pem", "key": "/etc/sensu/certs/gateway-key.pem", "url": "https://sensu.beta.example.com:2379", "api_version": "core/v2", "resource": "ClusterRoleBinding", "replication_interval_seconds": 30 } }
-
Run
sensuctl config view
and verify thatsensuctl
is configured to talk to agateway
cluster API. Reconfiguresensuctl
if needed. -
Save the
AlphaClusterRoleBindings
andBetaClusterRoleBindings
EtcdReplicator definitions to a file (for example,etcdreplicators.yml
oretcdreplicators.json
). -
Use
sensuctl create -f
to apply theAlphaClusterRoleBindings
andBetaClusterRoleBindings
EtcdReplicator definitions to thegateway
cluster:sensuctl create -f etcdreplicators.yml
sensuctl create -f etcdreplicators.json
-
Verify that the EtcdReplicator resource is working as expected: reconfigure the sensuctl backend URL to communicate with the
alpha
andbeta
clusters and run the following command for each:sensuctl cluster-role-binding info federation-viewer-readonly
The
federation-viewer-readonly
binding you created in the previous section should be listed in the output from each cluster:=== federation-viewer-readonly Name: federation-viewer-readonly Cluster Role: view Subjects: Users: federation-viewer
Register clusters
Clusters must be registered to become visible in the web UI. Each registered cluster must have a name and a list of one or more cluster member URLs corresponding to the backend REST API.
NOTE: Individual cluster resources may list the API URLs for a single stand-alone backend or multiple backends that are members of the same etcd cluster. Creating a cluster resource that lists multiple backends that do not belong to the same cluster will result in unexpected behavior.
Register a single cluster
With sensuctl
configured for the gateway
cluster, run sensuctl create
on the yaml or JSON below to register cluster alpha
:
api_version: federation/v1
type: Cluster
metadata:
name: alpha
spec:
api_urls:
- https://sensu.alpha.example.com:8080
{
"api_version": "federation/v1",
"type": "Cluster",
"metadata": {
"name": "alpha"
},
"spec": {
"api_urls": [
"https://sensu.alpha.example.com:8080"
]
}
}
Register additional clusters
With sensuctl
configured for gateway
cluster, run sensuctl create
on the yaml or JSON below to register an additional cluster and define the name as beta
:
api_version: federation/v1
type: Cluster
metadata:
name: beta
spec:
api_urls:
- https://sensu.beta.example.com:8080
{
"api_version": "federation/v1",
"type": "Cluster",
"metadata": {
"name": "beta"
},
"spec": {
"api_urls": [
"https://sensu.alpha.example.com:8080"
]
}
}
NOTE: When logging into the gateway
cluster web UI, any namespaces, entities, events, and other resources specific to that cluster will be labeled as local-cluster
.
Get a unified view of all your clusters in the web UI
After you create clusters using enterprise/federation/v1 API endpoints, you can log in to the gateway
Sensu web UI to view them as the federation-viewer
user.
Use the namespace switcher to change between namespaces across federated clusters:
Because the federation-viewer
user is granted only permissions provided by the built-in view
role, this user should be able to view all resources across all clusters but should not be able to make any changes.
If you haven’t changed the permissions of the default admin
user, that user should be able to view, create, delete, and update resources across all clusters.
What’s next
Learn more about configuring Sensu RBAC policies in the RBAC reference and one-way replication of RBAC resources in the etcd replicators reference.
Read the backend reference for details about the jwt-private-key-file
and jwt-public-key-file
attributes used to crypotgraphically vouch for the user’s identity in the JWT payload.