icinga2/doc/6.04-cluster.md

## <a id="cluster"></a> Cluster

An Icinga 2 cluster consists of two or more nodes and can reside on multiple
architectures. The base concept of Icinga 2 is the possibility to add additional
features using components. In case of a cluster setup you have to add the
cluster feature to all nodes. Before you start configuring the diffent nodes
it's necessary to setup the underlying communication layer based on SSL.

### <a id="certificate-authority-certificates"></a> Certificate Authority and Certificates

Icinga 2 comes with two scripts helping you to create CA and node certificates
for you Icinga 2 Cluster.

The first step is the creation of CA using the following command:

    icinga2-build-ca

Please make sure to export a variable containing an empty folder for the created
CA files:

    export ICINGA_CA="/root/icinga-ca"

In the next step you have to create a certificate and a key file for every node
using the following command:

    icinga2-build-key icinga-node-1

Please create a certificate and a key file for every node in the Icinga 2
Cluster and save the CA key in case you want to set up certificates for
additional nodes at a later date.

### <a id="enable-cluster-configuration"></a> Enable the Cluster Configuration

Until the cluster-component is moved into an independent feature you have to
enable the required libraries in the icinga2.conf configuration file:

    library "cluster"

### <a id="configure-clusterlistener-object"></a> Configure the ClusterListener Object

The ClusterListener needs to be configured on every node in the cluster with the
following settings:

  Configuration Setting    |Value
  -------------------------|------------------------------------
  ca_path                  | path to ca.crt file
  cert_path                | path to server certificate
  key_path                 | path to server key
  bind_port                | port for incoming and outgoing conns
  peers                    | array of all reachable nodes
  ------------------------- ------------------------------------

A sample config part can look like this:

    /**
     * Load cluster library and configure ClusterListener using certificate files
     */
    library "cluster"

    object ClusterListener "cluster" {
      ca_path = "/etc/icinga2/ca/ca.crt",
      cert_path = "/etc/icinga2/ca/icinga-node-1.crt",
      key_path = "/etc/icinga2/ca/icinga-node-1.key",

      bind_port = 8888,

      peers = [ "icinga-node-2" ]
    }

> **Note**
>
> The certificate files must be readable by the user Icinga 2 is running as. Also,
> the private key file should not be world-readable.

Peers configures the direction used to connect multiple nodes together. If have
a three node cluster consisting of

* node-1
* node-2
* node-3

and `node-3` is only reachable from `node-2`, you have to consider this in your
peer configuration.

### <a id="configure-cluster-endpoints"></a> Configure Cluster Endpoints

In addition to the configured port and hostname every endpoint can have specific
abilities to send configuration files to other nodes and limit the hosts allowed
to send configuration files.

  Configuration Setting    |Value
  -------------------------|------------------------------------
  host                     | hostname
  port                     | port
  accept_config            | defines all nodes allowed to send configs
  config_files             | defines all files to be send to that node - MUST BE AN ABSOLUTE PATH
  ------------------------- ------------------------------------

A sample config part can look like this:

    /**
     * Configure config master endpoint
     */

    object Endpoint "icinga-node-1" {
      host = "icinga-node-1.localdomain",
      port = 8888,
      config_files = ["/etc/icinga2/conf.d/*.conf"]
    }

If you update the configuration files on the configured file sender, it will
force a restart on all receiving nodes after validating the new config.

A sample config part for a config receiver endpoint can look like this:

    /**
     * Configure config receiver endpoint
     */

    object Endpoint "icinga-node-2" {
      host = "icinga-node-2.localdomain",
      port = 8888,
      accept_config = [ "icinga-node-1" ]
    }

By default these configuration files are saved in /var/lib/icinga2/cluster/config.

In order to load configuration files which were received from a remote Icinga 2
instance you will have to add the following include directive to your
`icinga2.conf` configuration file:

    include IcingaLocalStateDir + "/lib/icinga2/cluster/config/*/*"

### <a id="initial-cluster-sync"></a> Initial Cluster Sync

In order to make sure that all of your cluster nodes have the same state you will
have to pick one of the nodes as your initial "master" and copy its state file
to all the other nodes.

You can find the state file in `/var/lib/icinga2/icinga2.state`. Before copying
the state file you should make sure that all your cluster nodes are properly shut
down.


### <a id="assign-services-to-cluster-nodes"></a> Assign Services to Cluster Nodes

By default all services are distributed among the cluster nodes with the `Checker`
feature enabled.
If you require specific services to be only executed by one or more checker nodes
within the cluster, you must define `authorities` as additional service object
attribute. Required Endpoints must be defined as array.

    object Host "dmz-host1" inherits "generic-host" {
      services["dmz-oracledb"] = {
        templates = [ "generic-service" ],
        authorities = [ "icinga-node-1" ],
      }
    }

> **Tip**
>
> Most common usecase is building a classic Master-Slave-Setup. The master node
> does not have the `Checker` feature enabled, and the slave nodes are checking
> services based on their location, inheriting from a global service template
> defining the authorities.

### <a id="cluster-health-check"></a> Cluster Health Check

The Icinga 2 [ITL](#itl) ships an internal check command checking all configured
`EndPoints` in the cluster setup. The check result will become critical if
one or more configured nodes are not connected.

Example:

    object Host "icinga2a" inherits "generic-host" {
      services["cluster"] = {
        templates = [ "generic-service" ],
        check_interval = 1m,
        check_command = "cluster",
        authorities = [ "icinga2a" ]
      },
    }

> **Note**
>
> Each cluster node should execute its own local cluster health check to
> get an idea about network related connection problems from different
> point of views. Use the `authorities` attribute to assign the service
> check to the configured node.

### <a id="host-multiple-cluster-nodes"></a> Host With Multiple Cluster Nodes

Special scenarios might require multiple cluster nodes running on a single host.
By default Icinga 2 and its features will drop their runtime data below the prefix
`IcingaLocalStateDir`. By default packages will set that path to `/var`.
You can either set that variable as constant configuration
definition in [icinga2.conf](#icinga2-conf) or pass it as runtime variable to
the Icinga 2 daemon.

    # icinga2 -c /etc/icinga2/node1/icinga2.conf -DIcingaLocalStateDir=/opt/node1/var