icinga2/doc/6.04-cluster.md

Cluster

An Icinga 2 cluster consists of two or more nodes and can reside on multiple architectures. The base concept of Icinga 2 is the possibility to add additional features using components. In case of a cluster setup you have to add the cluster feature to all nodes. Before you start configuring the diffent nodes it's necessary to setup the underlying communication layer based on SSL.

Certificate Authority and Certificates

Icinga 2 comes with two scripts helping you to create CA and node certificates for you Icinga 2 Cluster.

The first step is the creation of CA using the following command:

icinga2-build-ca

Please make sure to export a variable containing an empty folder for the created CA files:

export ICINGA_CA="/root/icinga-ca"

In the next step you have to create a certificate and a key file for every node using the following command:

icinga2-build-key icinga-node-1

Please create a certificate and a key file for every node in the Icinga 2 Cluster and save the CA key in case you want to set up certificates for additional nodes at a later date.

Enable the Cluster Configuration

Until the cluster-component is moved into an independent feature you have to enable the required libraries in the icinga2.conf configuration file:

library "cluster"

Configure the ClusterListener Object

The ClusterListener needs to be configured on every node in the cluster with the following settings:

Configuration Setting	Value
ca_path	path to ca.crt file
cert_path	path to server certificate
key_path	path to server key
bind_port	port for incoming and outgoing conns
peers	array of all reachable nodes

---

A sample config part can look like this:

/**
 * Load cluster library and configure ClusterListener using certificate files
 */
library "cluster"

object ClusterListener "cluster" {
  ca_path = "/etc/icinga2/ca/ca.crt",
  cert_path = "/etc/icinga2/ca/icinga-node-1.crt",
  key_path = "/etc/icinga2/ca/icinga-node-1.key",

  bind_port = 8888,

  peers = [ "icinga-node-2" ]
}

Note

The certificate files must be readable by the user Icinga 2 is running as. Also, the private key file should not be world-readable.

Peers configures the direction used to connect multiple nodes together. If have a three node cluster consisting of

• node-1
• node-2
• node-3

and node-3 is only reachable from node-2, you have to consider this in your peer configuration.

Configure Cluster Endpoints

In addition to the configured port and hostname every endpoint can have specific abilities to send configuration files to other nodes and limit the hosts allowed to send configuration files.

Configuration Setting	Value
host	hostname
port	port
accept_config	defines all nodes allowed to send configs
config_files	defines all files to be send to that node - MUST BE AN ABSOLUTE PATH

---

A sample config part can look like this:

/**
 * Configure config master endpoint
 */

object Endpoint "icinga-node-1" {
  host = "icinga-node-1.localdomain",
  port = 8888,
  config_files = ["/etc/icinga2/conf.d/*.conf"]
}

If you update the configuration files on the configured file sender, it will force a restart on all receiving nodes after validating the new config.

A sample config part for a config receiver endpoint can look like this:

/**
 * Configure config receiver endpoint
 */

object Endpoint "icinga-node-2" {
  host = "icinga-node-2.localdomain",
  port = 8888,
  accept_config = [ "icinga-node-1" ]
}

By default these configuration files are saved in /var/lib/icinga2/cluster/config.

In order to load configuration files which were received from a remote Icinga 2 instance you will have to add the following include directive to your icinga2.conf configuration file:

include (IcingaLocalStateDir + "/lib/icinga2/cluster/config/*/*")

Initial Cluster Sync

In order to make sure that all of your cluster nodes have the same state you will have to pick one of the nodes as your initial "master" and copy its state file to all the other nodes.

You can find the state file in /var/lib/icinga2/icinga2.state. Before copying the state file you should make sure that all your cluster nodes are properly shut down.

Assign Services to Cluster Nodes

By default all services are distributed among the cluster nodes with the Checker feature enabled. If you require specific services to be only executed by one or more checker nodes within the cluster, you must define authorities as additional service object attribute. Required Endpoints must be defined as array.

object Host "dmz-host1" inherits "generic-host" {
  services["dmz-oracledb"] = {
    templates = [ "generic-service" ],
    authorities = [ "icinga-node-1" ],
  }
}

Tip

Most common usecase is building a classic Master-Slave-Setup. The master node does not have the Checker feature enabled, and the slave nodes are checking services based on their location, inheriting from a global service template defining the authorities.

Cluster Health Check

The Icinga 2 ITL ships an internal check command checking all configured EndPoints in the cluster setup. The check result will become critical if one or more configured nodes are not connected.

Example:

object Host "icinga2a" inherits "generic-host" {
  services["cluster"] = {
    templates = [ "generic-service" ],
    check_interval = 1m,
    check_command = "cluster",
    authorities = [ "icinga2a" ]
  },
}

Note

Each cluster node should execute its own local cluster health check to get an idea about network related connection problems from different point of views. Use the authorities attribute to assign the service check to the configured node.

Host With Multiple Cluster Nodes

Special scenarios might require multiple cluster nodes running on a single host. By default Icinga 2 and its features will drop their runtime data below the prefix IcingaLocalStateDir. By default packages will set that path to /var. You can either set that variable as constant configuration definition in icinga2.conf or pass it as runtime variable to the Icinga 2 daemon.

# icinga2 -c /etc/icinga2/node1/icinga2.conf -DIcingaLocalStateDir=/opt/node1/var