Multi-cluster visibility with federation

Sensu’s enterprise/federation/v1 API endpoints allow you to register external clusters, gain single-pane-of-glass visibility into the health of your infrastructure and services across multiple distinct Sensu instances within the web UI, and mirror your changes in one cluster to follower clusters. This is useful when you want to provide a single entry point for Sensu users who need to manage monitoring across multiple distinct physical data centers, cloud regions, or providers.

After you configure federation, you can also create, update, and delete clusters using sensuctl create, edit, and delete commands.

Federation is not enabled by default. You must create a cluster resource for the federation cluster and register it. Only cluster administrators can register a new cluster, but every user can query the list of clusters.

Complete federation of multiple Sensu instances relies on a combination of features:

Follow the example in this guide to configure these features. The example assumes you wish to federate three named Sensu clusters:

The gateway cluster will be the entry point for operators to manage Sensu resources in the alpha and beta clusters. This guide assumes a single sensu-backend in each cluster, but named clusters composed of multiple sensu-backends are supported.

This diagram depicts the federation relationship documented in this guide:

Complete the steps in this guide to browse events, entities, checks, and other resources in the gateway, alpha, and beta clusters from the gateway cluster web UI.

Requirements

Because federation depends on communication with multiple disparate clusters, working TLS is required for successful federated operation.

To ensure that cluster members can validate each other, certificates for each cluster member should include the IP addresses or hostnames specified in the values of sensu-backend etcd-advertise-client-urls, etcd-advertise-peer-urls, and etcd-initial-advertise-peer-urls parameters. In addition to the certificate’s Common Name (CN), Subject Alternative Names (SANs) are also honored for validation.

To continue with this guide, make sure you have the required TLS credentials in place:

• A PEM-formatted X.509 certificate and corresponding private key copied to each cluster member.
• A corresponding certificate authority (CA) certificate chain copied to each cluster member.

If you don’t have existing infrastructure for issuing certificates, read Generate certificates for our recommended self-signed certificate issuance process.

This prerequisite extends to configuring the following Sensu backend etcd parameters:

Configure shared token signing keys

Whether federated or standalone, Sensu backends issue JSON Web Tokens (JWTs) to users upon successful authentication. These tokens include a payload that describes the username and group affiliations. The payload is used to determine permissions based on the configured RBAC policy.

In a federation of Sensu backends, each backend needs to have the same public/private key pair. These asymmetric keys are used to crypotgraphically vouch for the user’s identity in the JWT payload. Using shared JWT keys enables clusters to grant users access to Sensu resources according to their local policies but without requiring user resources to be present uniformly across all clusters in the federation.

By default, a Sensu backend automatically generates an asymmetric key pair for signing JWTs and stores it in the etcd database. When configuring federation, you must generate keys as files on disk so they can be copied to all backends in the federation.

1. Use the openssl command line tool to generate a P-256 elliptic curve private key:

   openssl ecparam -genkey -name prime256v1 -noout -out jwt_private.pem

2. Generate a public key from the private key:

   openssl ec -in jwt_private.pem -pubout -out jwt_public.pem

3. Save the JWT keys in /etc/sensu/certs on each cluster backend.

4. Add the jwt-private-key-file and jwt-public-key-file attributes in /etc/sensu/backend.yml and specify the paths to the JWT private and public keys:

   jwt-private-key-file: /etc/sensu/certs/jwt_private.pem
   jwt-public-key-file: /etc/sensu/certs/jwt_public.pem

5. Restart the Sensu backend so that your settings take effect:

   sudo systemctl restart sensu-backend

Add a user and a cluster role binding

To test your configuration, provision a user and a cluster role binding in the gateway cluster.

1. Confirm that sensuctl is configured to communicate with the gateway cluster:

   sensuctl config view

   The response will list the active configuration:

   === Active Configuration
   API URL:   https://sensu.gateway.example.com:8080
   Namespace: default
   Format:    tabular
   Username:  admin

2. Create a federation-viewer user:

   sensuctl user create federation-viewer --interactive

3. When prompted for username and groups, press enter.

4. When prompted for password, enter a password for the federation-viewer user. Make a note of the password you entered — you’ll use it to log in to the web UI after you configure RBAC policy replication and registered clusters into your federation.

   This creates the following user:

   username: federation-viewer
   disabled: false

   {
     "username": "federation-viewer",
     "disabled": false
   }

5. Grant the federation-viewer user read-only access with a cluster role binding for the built-in view cluster role:

   sensuctl cluster-role-binding create federation-viewer-readonly --cluster-role=view --user=federation-viewer

   This command creates the following cluster role binding resource definition:

   ---
   type: ClusterRoleBinding
   api_version: core/v2
   metadata:
     created_by: admin
     name: federation-viewer-readonly
   spec:
     role_ref:
       name: view
       type: ClusterRole
     subjects:
     - name: federation-viewer
       type: User

   {
     "type": "ClusterRoleBinding",
     "api_version": "core/v2",
     "metadata": {
       "created_by": "admin",
       "name": "federation-viewer-readonly"
     },
     "spec": {
       "role_ref": {
         "name": "view",
         "type": "ClusterRole"
       },
       "subjects": [
         {
           "name": "federation-viewer",
           "type": "User"
         }
       ]
     }
   }

Create etcd replicators

Etcd replicators use the etcd make-mirror utility for one-way replication of Sensu RBAC policy resources. This allows you to centrally define RBAC policy on the gateway cluster and replicate RBAC resources to other clusters in the federation (alpha and beta), ensuring consistent permissions for Sensu users across multiple clusters via the gateway web UI.

1. Configure one etcd replicator per cluster for each RBAC policy resource, across all namespaces, for each backend in the federation.

   NOTE: Create a replicator for each resource type you want to replicate. Replicating namespace resources will not replicate the Sensu resources that belong to those namespaces.
   
   The etcd replicators reference includes examples you can follow for Role, RoleBinding, ClusterRole, and ClusterRoleBinding resources.

   In this example, the following etcd replicator resources will replicate ClusterRoleBinding resources from the gateway cluster to the two target clusters:

   ---
   api_version: federation/v1
   type: EtcdReplicator
   metadata:
     name: AlphaClusterRoleBindings
   spec:
     ca_cert: "/etc/sensu/certs/ca.pem"
     cert: "/etc/sensu/certs/gateway.pem"
     key: "/etc/sensu/certs/gateway-key.pem"
     url: https://sensu.alpha.example.com:2379
     api_version: core/v2
     resource: ClusterRoleBinding
     replication_interval_seconds: 30

   {
     "api_version": "federation/v1",
     "type": "EtcdReplicator",
     "metadata": {
       "name": "AlphaClusterRoleBindings"
     },
     "spec": {
       "ca_cert": "/etc/sensu/certs/ca.pem",
       "cert": "/etc/sensu/certs/gateway.pem",
       "key": "/etc/sensu/certs/gateway-key.pem",
       "url": "https://sensu.alpha.example.com:2379",
       "api_version": "core/v2",
       "resource": "ClusterRoleBinding",
       "replication_interval_seconds": 30
     }
   }

   ---
   api_version: federation/v1
   type: EtcdReplicator
   metadata:
     name: BetaClusterRoleBindings
   spec:
     ca_cert: "/etc/sensu/certs/ca.pem"
     cert: "/etc/sensu/certs/gateway.pem"
     key: "/etc/sensu/certs/gateway-key.pem"
     url: https://sensu.beta.example.com:2379
     api_version: core/v2
     resource: ClusterRoleBinding
     replication_interval_seconds: 30

   {
     "api_version": "federation/v1",
     "type": "EtcdReplicator",
     "metadata": {
       "name": "BetaClusterRoleBindings"
     },
     "spec": {
       "ca_cert": "/etc/sensu/certs/ca.pem",
       "cert": "/etc/sensu/certs/gateway.pem",
       "key": "/etc/sensu/certs/gateway-key.pem",
       "url": "https://sensu.beta.example.com:2379",
       "api_version": "core/v2",
       "resource": "ClusterRoleBinding",
       "replication_interval_seconds": 30
     }
   }

2. Run sensuctl config view and verify that sensuctl is configured to talk to a gateway cluster API. Reconfigure sensuctl if needed.

3. Save the AlphaClusterRoleBindings and BetaClusterRoleBindings EtcdReplicator definitions to a file (for example, etcdreplicators.yml or etcdreplicators.json).

4. Use sensuctl create -f to apply the AlphaClusterRoleBindings and BetaClusterRoleBindings EtcdReplicator definitions to the gateway cluster:

   sensuctl create -f etcdreplicators.yml

   sensuctl create -f etcdreplicators.json

5. Verify that the EtcdReplicator resource is working as expected: reconfigure the sensuctl backend URL to communicate with the alpha and beta clusters and run the following command for each:

   sensuctl cluster-role-binding info federation-viewer-readonly

   The federation-viewer-readonly binding you created in the previous section should be listed in the output from each cluster:

   === federation-viewer-readonly
   Name:         federation-viewer-readonly
   Cluster Role: view
   Subjects:
     Users:      federation-viewer

Register clusters

Clusters must be registered to become visible in the web UI. Each registered cluster must have a name and a list of one or more cluster member URLs corresponding to the backend REST API.

Register a single cluster

With sensuctl configured for the gateway cluster, run sensuctl create on the yaml or JSON below to register cluster alpha:

api_version: federation/v1
type: Cluster
metadata:
  name: alpha
spec:
  api_urls:
  - https://sensu.alpha.example.com:8080

{
  "api_version": "federation/v1",
  "type": "Cluster",
  "metadata": {
    "name": "alpha"
  },
  "spec": {
    "api_urls": [
      "https://sensu.alpha.example.com:8080"
    ]
  }
}

Register additional clusters

With sensuctl configured for gateway cluster, run sensuctl create on the yaml or JSON below to register an additional cluster and define the name as beta:

api_version: federation/v1
type: Cluster
metadata:
  name: beta
spec:
  api_urls:
  - https://sensu.beta.example.com:8080

{
  "api_version": "federation/v1",
  "type": "Cluster",
  "metadata": {
    "name": "beta"
  },
  "spec": {
    "api_urls": [
      "https://sensu.alpha.example.com:8080"
    ]
  }
}

Get a unified view of all your clusters in the web UI

After you create clusters using enterprise/federation/v1 API endpoints, you can log in to the gateway Sensu web UI to view them as the federation-viewer user. Use the namespace switcher to change between namespaces across federated clusters:

Because the federation-viewer user is granted only permissions provided by the built-in view role, this user should be able to view all resources across all clusters but should not be able to make any changes. If you haven’t changed the permissions of the default admin user, that user should be able to view, create, delete, and update resources across all clusters.

What’s next

Learn more about configuring Sensu RBAC policies in the RBAC reference and one-way replication of RBAC resources in the etcd replicators reference.

Read the backend reference for details about the jwt-private-key-file and jwt-public-key-file attributes used to crypotgraphically vouch for the user’s identity in the JWT payload.