icinga2/doc/19-technical-concepts.md

Technical Concepts

This chapter provides technical concepts and design insights into specific Icinga 2 components such as:

• Application
• Configuration
• Features
• Check Scheduler
• Checks
• Cluster
• TLS Network IO

Application

CLI Commands

The Icinga 2 application is managed with different CLI sub commands. daemon takes care about loading the configuration files, running the application as daemon, etc. Other sub commands allow to enable features, generate and request TLS certificates or enter the debug console.

The main entry point for each CLI command parses the command line parameters and then triggers the required actions.

daemon CLI command

This CLI command loads the configuration files, starting with icinga2.conf. The configuration compiler parses the file and detects additional file includes, constants, and any other DSL specific declaration.

At this stage, the configuration will already be checked against the defined grammar in the scanner, and custom object validators will also be checked.

If the user provided -C/--validate, the CLI command returns with the validation exit code.

When running as daemon, additional parameters are checked, e.g. whether this application was triggered by a reload, needs to daemonize with fork() involved and update the object's authority. The latter is important for HA-enabled cluster zones.

Configuration

Lexer

The lexer stage does not understand the DSL itself, it only maps specific character sequences into identifiers.

This allows Icinga to detect the beginning of a string with ", reading the following characters and determining the end of the string with again ".

Other parts covered by the lexer a escape sequences insides a string, e.g. "\"abc".

The lexer also identifiers logical operators, e.g. & or in, specific keywords like object, import, etc. and comment blocks.

Please check lib/config/config_lexer.ll for details.

Icinga uses Flex in the first stage.

Flex (The Fast Lexical Analyzer)

Flex is a fast lexical analyser generator. It is a tool for generating programs that perform pattern-matching on text. Flex is a free (but non-GNU) implementation of the original Unix lex program.

Parser

The parser stage puts the identifiers from the lexer into more context with flow control and sequences.

The following comparison is parsed into a left term, an operator and a right term.

x > 5

The DSL contains many elements which require a specific order, and sometimes only a left term for example.

The parser also takes care of parsing an object declaration for example. It already knows from the lexer that object marks the beginning of an object. It then expects a type string afterwards, and the object name - which can be either a string with double quotes or a previously defined constant.

An opening bracket { in this specific context starts the object scope, which also is stored for later scope specific variable access.

If there's an apply rule defined, this follows the same principle. The config parser detects the scope of an apply rule and generates Icinga 2 C++ code for the parsed string tokens.

assign where host.vars.sla == "24x7"

is parsed into an assign token identifier, and the string expression is compiled into a new ApplyExpression object.

The flow control inside the parser ensures that for example ignore where can only be defined when a previous assign where was given - or when inside an apply for rule.

Another example are specific object types which allow assign expression, specifically group objects. Others objects must throw a configuration error.

Please check lib/config/config_parser.yy for more details, and the language reference chapter for documented DSL keywords and sequences.

Icinga uses Bison as parser generator which reads a specification of a context-free language, warns about any parsing ambiguities, and generates a parser in C++ which reads sequences of tokens and decides whether the sequence conforms to the syntax specified by the grammar.

Compiler

The config compiler initializes the scanner inside the lexer stage.

The configuration files are parsed into memory from inside the daemon CLI command which invokes the config validation in ValidateConfigFiles(). This compiles the files into an AST expression which is executed.

At this stage, the expressions generate so-called "config items" which are a pre-stage of the later compiled object.

ConfigItem::CommitItems takes care of committing the items, and doing a rollback on failure. It also checks against matching apply rules from the previous run and generates statistics about the objects which can be seen by the config validation.

ConfigItem::CommitNewItems collects the registered types and items, and checks for a specific required order, e.g. a service object needs a host object first.

The following stages happen then:

• Commit: A workqueue then commits the items in a parallel fashion for this specific type. The object gets its name, and the AST expression is executed. It is then registered into the item into m_Object as reference.
• OnAllConfigLoaded: Special signal for each object to pre-load required object attributes, resolve group membership, initialize functions and timers.
• CreateChildObjects: Run apply rules for this specific type.
• CommitNewItems: Apply rules may generate new config items, this is to ensure that they again run through the stages.

Note that the items are now committed and the configuration is validated and loaded into memory. The final config objects are not yet activated though.

This only happens after the validation, when the application is about to be run with ConfigItem::ActivateItems.

Each item has an object created in m_Object which is checked in a loop. Again, the dependency order of activated objects is important here, e.g. logger features come first, then config objects and last the checker, api, etc. features. This is done by sorting the objects based on their type specific activation priority.

The following signals are triggered in the stages:

• PreActivate: Setting the active flag for the config object.
• Activate: Calls Start() on the object, sets the local HA authority and notifies subscribers that this object is now activated (e.g. for config updates in the DB backend).

References

• The Icinga Config Compiler: An Overview
• A parser/lexer/compiler for the Leonardo language
• I wrote a programming language. Here’s how you can, too.
• http://onoffswitch.net/building-a-custom-lexer/
• Writing an Interpreter with Lex, Yacc, and Memphis
• Flex
• GNU Bison

Features

Features are implemented in specific libraries and can be enabled using CLI commands.

Features either write specific data or receive data.

Examples for writing data: DB IDO, Graphite, InfluxDB. GELF, etc. Examples for receiving data: REST API, etc.

The implementation of features makes use of existing libraries and functionality. This makes the code more abstract, but shorter and easier to read.

Features register callback functions on specific events they want to handle. For example the GraphiteWriter feature subscribes to new CheckResult events.

Each time Icinga 2 receives and processes a new check result, this event is triggered and forwarded to all subscribers.

The GraphiteWriter feature calls the registered function and processes the received data. Features which connect Icinga 2 to external interfaces normally parse and reformat the received data into an applicable format.

Since this check result signal is blocking, many of the features include a work queue with asynchronous task handling.

The GraphiteWriter uses a TCP socket to communicate with the carbon cache daemon of Graphite. The InfluxDBWriter is instead writing bulk metric messages to InfluxDB's HTTP API, similar to Elasticsearch.

Check Scheduler

The check scheduler starts a thread which loops forever. It waits for check events being inserted into m_IdleCheckables.

If the current pending check event number is larger than the configured max concurrent checks, the thread waits up until it there's slots again.

In addition, further checks on enabled checks, check periods, etc. are performed. Once all conditions have passed, the next check timestamp is calculated and updated. This also is the timestamp where Icinga expects a new check result ("freshness check").

The object is removed from idle checkables, and inserted into the pending checkables list. This can be seen via REST API metrics for the checker component feature as well.

The actual check execution happens asynchronously using the application's thread pool.

Once the check returns, it is removed from pending checkables and again inserted into idle checkables. This ensures that the scheduler takes this checkable event into account in the next iteration.

Start

When checkable objects get activated during the startup phase, the checker feature registers a handler for this event. This is due to the fact that the checker feature is fully optional, and e.g. not used on command endpoint clients.

Whenever such an object activation signal is triggered, Icinga 2 checks whether it is authoritative for this object. This means that inside an HA enabled zone with two endpoints, only non-paused checkable objects are actively inserted into the idle checkable list for the check scheduler.

Initial Check

When a new checkable object (host or service) is initially added to the configuration, Icinga 2 performs the following during startup:

• Checkable::Start() is called and calculates the first check time
• With a spread delta, the next check time is actually set.

If the next check should happen within a time frame of 60 seconds, Icinga 2 calculates a delta from a random value. The minimum of check_interval and 60 seconds is used as basis, multiplied with a random value between 0 and 1.

In the best case, this check gets immediately executed after application start. The worst case scenario is that the check is scheduled 60 seconds after start the latest.

The reasons for delaying and spreading checks during startup is that the application typically needs more resources at this time (cluster connections, feature warmup, initial syncs, etc.). Immediate check execution with thousands of checks could lead into performance problems, and additional events for each received check results.

Therefore the initial check window is 60 seconds on application startup, random seed for all checkables. This is not predictable over multiple restarts for specific checkable objects, the delta changes every time.

Scheduling Offset

There's a high chance that many checkable objects get executed at the same time and interval after startup. The initial scheduling spreads that a little, but Icinga 2 also attempts to ensure to keep fixed intervals, even with high check latency.

During startup, Icinga 2 calculates the scheduling offset from a random number:

• Checkable::Checkable() calls SetSchedulingOffset() with Utility::Random()
• The offset is a pseudo-random integral value between 0 and RAND_MAX.

Whenever the next check time is updated with Checkable::UpdateNextCheck(), the scheduling offset is taken into account.

Depending on the state type (SOFT or HARD), either the retry_interval or check_interval is used. If the interval is greater than 1 second, the time adjustment is calculated in the following way:

now * 100 + offset divided by interval * 100, using the remainder (that's what fmod() is for) and dividing this again onto base 100.

Example: offset is 6500, interval 300, now is 1542190472.

1542190472 * 100 + 6500 = 154219053714
300 * 100 = 30000
154219053714 / 30000 = 5140635.1238

(5140635.1238 - 5140635.0) * 30000 = 3714
3714 / 100 = 37.14

37.15 seconds as an offset would be far too much, so this is again used as a calculation divider for the real offset with the base of 5 times the actual interval.

Again, the remainder is calculated from the offset and interval * 5. This is divided onto base 100 again, with an additional 0.5 seconds delay.

Example: offset is 6500, interval 300.

6500 / 300 = 21.666666666666667
(21.666666666666667 - 21.0) * 300 = 200
200 / 100 = 2
2 + 0.5 = 2.5

The minimum value between the first adjustment and the second offset calculation based on the interval is taken, in the above example 2.5 wins.

The actual next check time substracts the adjusted time from the future interval addition to provide a more widespread scheduling time among all checkable objects.

nextCheck = now - adj + interval

You may ask, what other values can happen with this offset calculation. Consider calculating more examples with different interval settings.

Example: offset is 34567, interval 60, now is 1542190472.

1542190472 * 100 + 34567 = 154219081767
60 * 100 = 6000
154219081767 / 6000 = 25703180.2945
(25703180.2945 - 25703180.0) * 6000 / 100 = 17.67

34567 / 60 = 576.116666666666667
(576.116666666666667 - 576.0) * 60 / 100 + 0.5 = 1.2

1m interval starts at now + 1.2s.

Example: offset is 12345, interval 86400, now is 1542190472.

1542190472 * 100 + 12345 = 154219059545
86400 * 100 = 8640000
154219059545 / 8640000 = 17849.428188078703704
(17849.428188078703704 - 17849) * 8640000 = 3699545
3699545 / 100 = 36995.45

12345 / 86400 = 0.142881944444444
0.142881944444444 * 86400 / 100 + 0.5 = 123.95

1d interval starts at now + 2m4s.

Note

In case you have a better algorithm at hand, feel free to discuss this in a PR on GitHub. It needs to fulfill two things: 1) spread and shuffle execution times on each next_check update 2) not too narrowed window for both long and short intervals Application startup and initial checks need to be handled with care in a slightly different fashion.

When SetNextCheck() is called, there are signals registered. One of them sits inside the CheckerComponent class whose handler CheckerComponent::NextCheckChangedHandler() deletes/inserts the next check event from the scheduling queue. This basically is a list with multiple indexes with the keys for scheduling info and the object.

Checks

Check Latency and Execution Time

Each check command execution logs the start and end time where Icinga 2 (and the end user) is able to calculate the plugin execution time from it.

GetExecutionEnd() - GetExecutionStart()

The higher the execution time, the higher the command timeout must be set. Furthermore users and developers are encouraged to look into plugin optimizations to minimize the execution time. Sometimes it is better to let an external daemon/script do the checks and feed them back via REST API.

Icinga 2 stores the scheduled start and end time for a check. If the actual check execution time differs from the scheduled time, e.g. due to performance problems or limited execution slots (concurrent checks), this value is stored and computed from inside the check result.

The difference between the two deltas is called check latency.

(GetScheduleEnd() - GetScheduleStart()) - CalculateExecutionTime()

Severity

The severity attribute is introduced with Icinga v2.11 and provides a bit mask calculated value from specific checkable object states.

The severity value is pre-calculated for visualization interfaces such as Icinga Web which sorts the problem dashboard by severity by default.

The higher the severity number is, the more important the problem is.

Flags:

/**
 * Severity Flags
 *
 * @ingroup icinga
 */
enum SeverityFlag
{
	SeverityFlagDowntime = 1,
	SeverityFlagAcknowledgement = 2,
	SeverityFlagHostDown = 4,
	SeverityFlagUnhandled = 8,
	SeverityFlagPending = 16,
	SeverityFlagWarning = 32,
	SeverityFlagUnknown = 64,
	SeverityFlagCritical = 128,
};

Host:

	/* OK/Warning = Up, Critical/Unknown = Down */
	if (!HasBeenChecked())
		severity |= SeverityFlagPending;
	else if (state == ServiceUnknown)
		severity |= SeverityFlagCritical;
	else if (state == ServiceCritical)
		severity |= SeverityFlagCritical;

	if (IsInDowntime())
		severity |= SeverityFlagDowntime;
	else if (IsAcknowledged())
		severity |= SeverityFlagAcknowledgement;
	else
		severity |= SeverityFlagUnhandled;

Service:

	if (!HasBeenChecked())
		severity |= SeverityFlagPending;
	else if (state == ServiceWarning)
		severity |= SeverityFlagWarning;
	else if (state == ServiceUnknown)
		severity |= SeverityFlagUnknown;
	else if (state == ServiceCritical)
		severity |= SeverityFlagCritical;

	if (IsInDowntime())
		severity |= SeverityFlagDowntime;
	else if (IsAcknowledged())
		severity |= SeverityFlagAcknowledgement;
	else if (m_Host->GetProblem())
		severity |= SeverityFlagHostDown;
	else
		severity |= SeverityFlagUnhandled;

Cluster

Communication

Icinga 2 uses its own certificate authority (CA) by default. The public and private CA keys can be generated on the signing master.

Each node certificate must be signed by the private CA key.

Note: The following description uses parent node and child node. This also applies to nodes in the same cluster zone.

During the connection attempt, an SSL handshake is performed. If the public certificate of a child node is not signed by the same CA, the child node is not trusted and the connection will be closed.

If the SSL handshake succeeds, the parent node reads the certificate's common name (CN) of the child node and looks for a local Endpoint object name configuration.

If there is no Endpoint object found, further communication (runtime and config sync, etc.) is terminated.

The child node also checks the CN from the parent node's public certificate. If the child node does not find any local Endpoint object name configuration, it will not trust the parent node.

Both checks prevent accepting cluster messages from an untrusted source endpoint.

If an Endpoint match was found, there is one additional security mechanism in place: Endpoints belong to a Zone hierarchy.

Several cluster messages can only be sent "top down", others like check results are allowed being sent from the child to the parent node.

Once this check succeeds the cluster messages are exchanged and processed.

CSR Signing

In order to make things easier, Icinga 2 provides built-in methods to allow child nodes to request a signed certificate from the signing master.

Icinga 2 v2.8 introduces the possibility to request certificates from indirectly connected nodes. This is required for multi level cluster environments with masters, satellites and clients.

CSR Signing in general starts with the master setup. This step ensures that the master is in a working CSR signing state with:

• public and private CA key in /var/lib/icinga2/ca
• private TicketSalt constant defined inside the api feature
• Cluster communication is ready and Icinga 2 listens on port 5665

The child node setup which is run with CLI commands will now attempt to connect to the parent node. This is not necessarily the signing master instance, but could also be a parent satellite node.

During this process the child node asks the user to verify the parent node's public certificate to prevent MITM attacks.

There are two methods to request signed certificates:

• Add the ticket into the request. This ticket was generated on the master beforehand and contains hashed details for which client it has been created. The signing master uses this information to automatically sign the certificate request.

• Do not add a ticket into the request. It will be sent to the signing master which stores the pending request. Manual user interaction with CLI commands is necessary to sign the request.

The certificate request is sent as pki::RequestCertificate cluster message to the parent node.

If the parent node is not the signing master, it stores the request in /var/lib/icinga2/certificate-requests and forwards the cluster message to its parent node.

Once the message arrives on the signing master, it first verifies that the sent certificate request is valid. This is to prevent unwanted errors or modified requests from the "proxy" node.

After verification, the signing master checks if the request contains a valid signing ticket. It hashes the certificate's common name and compares the value to the received ticket number.

If the ticket is valid, the certificate request is immediately signed with CA key. The request is sent back to the client inside a pki::UpdateCertificate cluster message.

If the child node was not the certificate request origin, it only updates the cached request for the child node and send another cluster message down to its child node (e.g. from a satellite to a client).

If no ticket was specified, the signing master waits until the ca sign CLI command manually signed the certificate.

Note

Push notifications for manual request signing is not yet implemented (TODO).

Once the child node reconnects it synchronizes all signed certificate requests. This takes some minutes and requires all nodes to reconnect to each other.

CSR Signing: Clients without parent connection

There is an additional scenario: The setup on a child node does not necessarily need a connection to the parent node.

This mode leaves the node in a semi-configured state. You need to manually copy the master's public CA key into /var/lib/icinga2/certs/ca.crt on the client before starting Icinga 2.

The parent node needs to actively connect to the child node. Once this connections succeeds, the child node will actively request a signed certificate.

The update procedure works the same way as above.

High Availability

General high availability is automatically enabled between two endpoints in the same cluster zone.

This requires the same configuration and enabled features on both nodes.

HA zone members trust each other and share event updates as cluster messages. This includes for example check results, next check timestamp updates, acknowledgements or notifications.

This ensures that both nodes are synchronized. If one node goes away, the remaining node takes over and continues as normal.

High Availability: Object Authority

Cluster nodes automatically determine the authority for configuration objects. By default, all config objects are set to HARunEverywhere and as such the object authority is true for any config object on any instance.

Specific objects can override and influence this setting, e.g. with HARunOnce instead prior to config object activation.

This is done when the daemon starts and in a regular interval inside the ApiListener class, specifically calling ApiListener::UpdateObjectAuthority().

The algorithm works like this:

• Determine whether this instance is assigned to a local zone and endpoint.
• Collects all endpoints in this zone if they are connected.
• If there's two endpoints, but only us seeing ourselves and the application start is less than 60 seconds in the past, do nothing (wait for cluster reconnect to take place, grace period).
• Sort the collected endpoints by name.
• Iterate over all config types and their respective objects
• Ignore !active objects
• Ignore objects which are !HARunOnce. This means, they can run multiple times in a zone and don't need an authority update.
• If this instance doesn't have a local zone, set authority to true. This is for non-clustered standalone environments where everything belongs to this instance.
• Calculate the object authority based on the connected endpoint names.
• Set the authority (true or false)

The object authority calculation works "offline" without any message exchange. Each instance alculates the SDBM hash of the config object name, puts that in contrast modulo the connected endpoints size. This index is used to lookup the corresponding endpoint in the connected endpoints array, including the local endpoint. Whether the local endpoint is equal to the selected endpoint, or not, this sets the authority to true or false.

authority = endpoints[Utility::SDBM(object->GetName()) % endpoints.size()] == my_endpoint;

ConfigObject::SetAuthority(bool authority) triggers the following events:

• Authority is true and object now paused: Resume the object and set paused to false.
• Authority is false, object not paused: Pause the object and set paused to true.

This results in activated but paused objects on one endpoint. You can verify that by querying the paused attribute for all objects via REST API or debug console on both endpoints.

Endpoints inside a HA zone calculate the object authority independent from each other. This object authority is important for selected features explained below.

Since features are configuration objects too, you must ensure that all nodes inside the HA zone share the same enabled features. If configured otherwise, one might have a checker feature on the left node, nothing on the right node. This leads to late check results because one half is not executed by the right node which holds half of the object authorities.

By default, features are enabled to "Run-Everywhere". Specific features which support HA awareness, provide the enable_ha configuration attribute. When enable_ha is set to true (usually the default), "Run-Once" is set and the feature pauses on one side.

vim /etc/icinga2/features-enabled/graphite.conf

object GraphiteWriter "graphite" {
  ...
  enable_ha = true
}

Once such a feature is paused, there won't be any more event handling, e.g. the Elasticsearch feature won't process any checkresults nor write to the Elasticsearch REST API.

When the cluster connection drops, the feature configuration object is updated with the new object authority by the ApiListener timer and resumes its operation. You can see that by grepping the log file for resumed and paused.

[2018-10-24 13:28:28 +0200] information/GraphiteWriter: 'g-ha' paused.

[2018-10-24 13:28:28 +0200] information/GraphiteWriter: 'g-ha' resumed.

Specific features with HA capabilities are explained below.

High Availability: Checker

The checker feature only executes checks for Checkable objects (Host, Service) where it is authoritative.

That way each node only executes checks for a segment of the overall configuration objects.

The cluster message routing ensures that all check results are synchronized to nodes which are not authoritative for this configuration object.

High Availability: Notifications

The notification feature only sends notifications for Notification objects where it is authoritative.

That way each node only executes notifications for a segment of all notification objects.

Notified users and other event details are synchronized throughout the cluster. This is required if for example the DB IDO feature is active on the other node.

High Availability: DB IDO

If you don't have HA enabled for the IDO feature, both nodes will write their status and historical data to their own separate database backends.

In order to avoid data separation and a split view (each node would require its own Icinga Web 2 installation on top), the high availability option was added to the DB IDO feature. This is enabled by default with the enable_ha setting.

This requires a central database backend. Best practice is to use a MySQL cluster with a virtual IP.

Both Icinga 2 nodes require the connection and credential details configured in their DB IDO feature.

During startup Icinga 2 calculates whether the feature configuration object is authoritative on this node or not. The order is an alpha-numeric comparison, e.g. if you have master1 and master2, Icinga 2 will enable the DB IDO feature on master2 by default.

If the connection between endpoints drops, the object authority is re-calculated.

In order to prevent data duplication in a split-brain scenario where both nodes would write into the same database, there is another safety mechanism in place.

The split-brain decision which node will write to the database is calculated from a quorum inside the programstatus table. Each node verifies whether the endpoint_name column is not itself on database connect. In addition to that the DB IDO feature compares the last_update_time column against the current timestamp plus the configured failover_timeout offset.

That way only one active DB IDO feature writes to the database, even if they are not currently connected in a cluster zone. This prevents data duplication in historical tables.

Health Checks

cluster-zone

This built-in check provides the possibility to check for connectivity between zones.

If you for example need to know whether the master zone is connected and processing messages with the child zone called satellite in this example, you can configure the cluster-zone check as new service on all master zone hosts.

vim /etc/zones.d/master/host1.conf

object Service "cluster-zone-satellite" {
  check_command = "cluster-zone"
  host_name = "host1"

  vars.cluster_zone = "satellite"
}

The check itself changes to NOT-OK if one or more child endpoints in the child zone are not connected to parent zone endpoints.

In addition to the overall connectivity check, the log lag is calculated based on the to-be-sent replay log. Each instance stores that for its configured endpoint objects.

This health check iterates over the target zone (cluster_zone) and their endpoints.

The log lag is greater than zero if

• the replay log synchronization is in progress and not yet finished or
• the endpoint is not connected, and no replay log sync happened (obviously).

The final log lag value is the worst value detected. If satellite1 has a log lag of 1.5 and satellite2 only has 0.5, the computed value will be 1.5..

You can control the check state by using optional warning and critical thresholds for the log lag value.

If this service exists multiple times, e.g. for each master host object, the log lag may differ based on the execution time. This happens for example on restart of an instance when the log replay is in progress and a health check is executed at different times. If the endpoint is not connected, both master instances may have saved a different log replay position from the last synchronisation.

The lag value is returned as performance metric key slave_lag.

Icinga 2 v2.9+ adds more performance metrics for these values:

• last_messages_sent and last_messages_received as UNIX timestamp
• sum_messages_sent_per_second and sum_messages_received_per_second
• sum_bytes_sent_per_second and sum_bytes_received_per_second

TLS Network IO

TLS Connection Handling

Icinga supports two connection directions, controlled via the host attribute inside the Endpoint objects:

• Outgoing connection attempts
• Incoming connection handling

Once the connection is established, higher layers can exchange JSON-RPC and HTTP messages. It doesn't matter which direction these message go.

This offers a big advantage over single direction connections, just like polling via HTTP only. Also, connections are kept alive as long as data is transmitted.

When the master connects to the child zone member(s), this requires more resources there. Keep this in mind when endpoints are not reachable, the TCP timeout blocks other resources. Moving a satellite zone in the middle between masters and agents/clients helps to split the tasks - the master processes and stores data, deploys configuration and serves the API. The satellites schedule the checks, connect to the agents/clients and receive check results.

Agents/Clients can also connect to the parent endpoints - be it a master or a satellite. This is the preferred way out of a DMZ, and also reduces the overhead with connecting to e.g. 2000 agents on the master. You can benchmark this when TCP connections are broken and timeouts are encountered.

Master Processes Incoming Connection

• The node starts a new ApiListener, this invokes AddListener()
  • Setup SSL Context
  • Initialize global I/O engine and create a TCP acceptor
  • Resolve bind host/port (optional)
  • Listen on IPv4 and IPv6
  • Re-use socket address and port
  • Listen on port 5665 with INT_MAX possible sockets
• Spawn a new Coroutine which listens for new incoming connections as 'TCP server' pattern
  • Accept new connections asynchronously
  • Spawn a new Coroutine which handles the new client connection in a different context, Role: Server

Master Connects Outgoing

• The node starts a timer in a 10 seconds interval with ApiReconnectTimerHandler() as callback
  • Loop over all configured zones, exclude global zones and not direct parent/child zones
  • Get the endpoints configured in the zones, exclude: local endpoint, no 'host' attribute, already connected or in progress
  • Call AddConnection()
• Spawn a new Coroutine after making the SSL context
  • Use the global I/O engine for socket I/O
  • Create TLS stream
  • Connect to endpoint host/port details
  • Handle the client connection, Role: Client

TLS Handshake

• Create a TLS connection in sslConn and perform an asynchronous TLS handshake
• Get the peer certificate
• Verify the presented certificate: ssl::verify_peer and ssl::verify_client_once
• Get the certificate CN and compare it against the endpoint name - if not matching, return and close the connection

Data Exchange

Everything runs through TLS, we don't use any "raw" connections nor plain message handling.

HTTP and JSON-RPC messages share the same port and API, so additional handling is required.

On a new connection and successful TLS handshake, the first byte is read. This either is a JSON-RPC message in Netstring format starting with a number, or plain HTTP.

HTTP/1.1

2:{}

Depending on this, ClientJsonRpc or ClientHttp are assigned.

JSON-RPC:

• Create a new JsonRpcConnection object
  • When the endpoint object is configured, spawn a Coroutine which takes care of syncing the client (file and runtime config, replay log, etc.)
  • No endpoint treats this connection as anonymous client, with a configurable limit. This client may send a CSR signing request for example.
  • Start the JsonRpcConnection - this spawns Coroutines to HandleIncomingMessages, WriteOutgoingMessages, HandleAndWriteHeartbeats and CheckLiveness

HTTP:

• Create a new HttpServerConnection
  • Start the HttpServerConnection - this spawns Coroutines to ProcessMessages and CheckLiveness

All the mentioned Coroutines run asynchronously using the global I/O engine's context. More details on this topic can be found in this blogpost.

The lower levels of context switching and sharing or event polling are hidden in Boost ASIO, Beast, Coroutine and Context libraries.

Data Exchange: Coroutines and I/O Engine

Light-weight and fast operations such as connection handling or TLS handshakes are performed in the default IoBoundWorkSlot pool inside the I/O engine.

The I/O engine has another pool available: CpuBoundWork.

This is used for processing CPU intensive tasks, such as handling a HTTP request. Depending on the available CPU cores, this is limited to std::thread::hardware_concurrency() * 3u / 2u.

1 core * 3 / 2 = 1
2 cores * 3 / 2 = 3
8 cores * 3 / 2 = 12
16 cores * 3 / 2 = 24

The I/O engine itself is used with all network I/O in Icinga, not only the cluster and the REST API. Features such as Graphite, InfluxDB, etc. also consume its functionality.

There are 2 * CPU cores threads available which run the event loop in the I/O engine. This polls the I/O service with m_IoService.run(); and triggers an asynchronous event progress for waiting coroutines.

JSON-RPC Message API

The JSON-RPC message API is not a public API for end users. In case you want to interact with Icinga, use the REST API.

This section describes the internal cluster messages exchanged between endpoints.

Tip

Debug builds with icinga2 daemon -DInternal.DebugJsonRpc=1 unveils the JSON-RPC messages.

Registered Handler Functions

Functions by example:

Event Sender: Checkable::OnNewCheckResult

On<xyz>.connect(&xyzHandler)

Event Receiver (Client): CheckResultAPIHandler in REGISTER_APIFUNCTION

<xyz>APIHandler()

Messages

icinga::Hello

Location: apilistener.cpp

Message Body

Key	Value
jsonrpc	2.0
method	icinga::Hello
params	Dictionary

Params

Currently empty.

Functions

Event Sender: When a new client connects in NewClientHandlerInternal(). Event Receiver: HelloAPIHandler

Permissions

None, this is a required message.

event::Heartbeat

Location: jsonrpcconnection-heartbeat.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::Heartbeat
params	Dictionary

Params

Key	Type	Description
timeout	Number	Heartbeat timeout, sender sets 120s.

Functions

Event Sender: JsonRpcConnection::HeartbeatTimerHandler Event Receiver: HeartbeatAPIHandler

Both sender and receiver exchange this heartbeat message. If the sender detects that a client endpoint hasn't sent anything in the updated timeout span, it disconnects the client. This is to avoid stale connections with no message processing.

Permissions

None, this is a required message.

event::CheckResult

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::CheckResult
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
cr	Serialized CR	Check result

Functions

Event Sender: Checkable::OnNewCheckResult Event Receiver: CheckResultAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Hosts/services do not exist
• Origin is a remote command endpoint different to the configured, and whose zone is not allowed to access this checkable.

event::SetNextCheck

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::SetNextCheck
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
next_check	Timestamp	Next scheduled time as UNIX timestamp.

Functions

Event Sender: Checkable::OnNextCheckChanged Event Receiver: NextCheckChangedAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Checkable does not exist.
• Origin endpoint's zone is not allowed to access this checkable.

event::SetNextNotification

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::SetNextNotification
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
notification	String	Notification name
next_notification	Timestamp	Next scheduled notification time as UNIX timestamp.

Functions

Event Sender: Notification::OnNextNotificationChanged Event Receiver: NextNotificationChangedAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Notification does not exist.
• Origin endpoint's zone is not allowed to access this checkable.

event::SetForceNextCheck

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::SetForceNextCheck
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
forced	Boolean	Forced next check (execute now)

Functions

Event Sender: Checkable::OnForceNextCheckChanged Event Receiver: ForceNextCheckChangedAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Checkable does not exist.
• Origin endpoint's zone is not allowed to access this checkable.

event::SetForceNextNotification

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::SetForceNextNotification
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
forced	Boolean	Forced next check (execute now)

Functions

Event Sender: Checkable::SetForceNextNotification Event Receiver: ForceNextNotificationChangedAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Checkable does not exist.
• Origin endpoint's zone is not allowed to access this checkable.

event::SetAcknowledgement

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::SetAcknowledgement
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
author	String	Acknowledgement author name.
comment	String	Acknowledgement comment content.
acktype	Number	Acknowledgement type (0=None, 1=Normal, 2=Sticky)
notify	Boolean	Notification should be sent.
persistent	Boolean	Whether the comment is persistent.
expiry	Timestamp	Optional expire time as UNIX timestamp.

Functions

Event Sender: Checkable::OnForceNextCheckChanged Event Receiver: ForceNextCheckChangedAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Checkable does not exist.
• Origin endpoint's zone is not allowed to access this checkable.

event::ClearAcknowledgement

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::ClearAcknowledgement
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name

Functions

Event Sender: Checkable::OnAcknowledgementCleared Event Receiver: AcknowledgementClearedAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Checkable does not exist.
• Origin endpoint's zone is not allowed to access this checkable.

event::SendNotifications

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::SendNotifications
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
cr	Serialized CR	Check result
type	Number	enum NotificationType, same as types for notification objects.
author	String	Author name
text	String	Notification text

Functions

Event Sender: Checkable::OnNotificationsRequested Event Receiver: SendNotificationsAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Checkable does not exist.
• Origin endpoint's zone the same as the receiver. This binds notification messages to the HA zone.

event::NotificationSentUser

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::NotificationSentUser
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
notification	String	Notification name.
user	String	Notified user name.
type	Number	enum NotificationType, same as types in Notification objects.
cr	Serialized CR	Check result.
author	String	Notification author (for specific types)
text	String	Notification text (for specific types)
command	String	Notification command name.

Functions

Event Sender: Checkable::OnNotificationSentToUser Event Receiver: NotificationSentUserAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Checkable does not exist.
• Origin endpoint's zone the same as the receiver. This binds notification messages to the HA zone.

event::NotificationSentToAllUsers

Location: clusterevents.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::NotificationSentToAllUsers
params	Dictionary

Params

Key	Type	Description
host	String	Host name
service	String	Service name
notification	String	Notification name.
users	Array of String	Notified user names.
type	Number	enum NotificationType, same as types in Notification objects.
cr	Serialized CR	Check result.
author	String	Notification author (for specific types)
text	String	Notification text (for specific types)
last_notification	Timestamp	Last notification time as UNIX timestamp.
next_notification	Timestamp	Next scheduled notification time as UNIX timestamp.
notification_number	Number	Current notification number in problem state.
last_problem_notification	Timestamp	Last problem notification time as UNIX timestamp.
no_more_notifications	Boolean	Whether to send future notifications when this notification becomes active on this HA node.

Functions

Event Sender: Checkable::OnNotificationSentToAllUsers Event Receiver: NotificationSentToAllUsersAPIHandler

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Checkable does not exist.
• Origin endpoint's zone the same as the receiver. This binds notification messages to the HA zone.

event::ExecuteCommand

Location: clusterevents-check.cpp and checkable-check.cpp

Message Body

Key	Value
jsonrpc	2.0
method	event::ExecuteCommand
params	Dictionary

Params

Key	Type	Description
host	String	Host name.
service	String	Service name.
command_type	String	check_command or event_command.
command	String	CheckCommand or EventCommand name.
macros	Dictionary	Command arguments as key/value pairs for remote execution.

Functions

Event Sender: This gets constructed directly in Checkable::ExecuteCheck() or Checkable::ExecuteEventHandler() when a remote command endpoint is configured.

• Get{CheckCommand,EventCommand}()->Execute() simulates an execution and extracts all command arguments into the macro dictionary (inside lib/methods tasks).
• When the endpoint is connected, the message is constructed and sent directly.
• When the endpoint is not connected and not syncing replay logs and 5m after application start, generate an UNKNOWN check result for the user ("not connected").

Event Receiver: ExecuteCommandAPIHandler

Special handling, calls ClusterEvents::EnqueueCheck() for command endpoint checks. This function enqueues check tasks into a queue which is controlled in RemoteCheckThreadProc().

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Origin endpoint's zone is not a parent zone of the receiver endpoint.
• accept_commands = false in the api feature configuration sends back an UNKNOWN check result to the sender.

The receiver constructs a virtual host object and looks for the local CheckCommand object.

Returns UNKNWON as check result to the sender

• when the CheckCommand object does not exist.
• when there was an exception triggered from check execution, e.g. the plugin binary could not be executed or similar.

The returned messages are synced directly to the sender's endpoint, no cluster broadcast.

Note: EventCommand errors are just logged on the remote endpoint.

config::Update

Location: apilistener-filesync.cpp

Message Body

Key	Value
jsonrpc	2.0
method	config::Update
params	Dictionary

Params

Key	Type	Description
update	Dictionary	Config file paths and their content.
update_v2	Dictionary	Additional meta config files introduced in 2.4+ for compatibility reasons.

Functions

Event Sender: SendConfigUpdate() called in ApiListener::SyncClient() when a new client endpoint connects. Event Receiver: ConfigUpdateHandler reads the config update content and stores them in /var/lib/icinga2/api. When it detects a configuration change, the function requests and application restart.

Permissions

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• The origin sender is not in a parent zone of the receiver.
• api feature does not accept config.

Config updates will be ignored when:

• The zone is not configured on the receiver endpoint.
• The zone is authoritative on this instance (this only happens on a master which has /etc/icinga2/zones.d populated, and prevents sync loops)

config::UpdateObject

Location: apilistener-configsync.cpp

Message Body

Key	Value
jsonrpc	2.0
method	config::UpdateObject
params	Dictionary

Params

Key	Type	Description
name	String	Object name.
type	String	Object type name.
version	Number	Object version.
config	String	Config file content for _api packages.
modified_attributes	Dictionary	Modified attributes at runtime as key value pairs.
original_attributes	Array	Original attributes as array of keys.

Functions

Event Sender: Either on client connect (full sync), or runtime created/updated object

ApiListener::SendRuntimeConfigObjects() gets called when a new endpoint is connected and runtime created config objects need to be synced. This invokes a call to UpdateConfigObject() to only sync this JsonRpcConnection client.

ConfigObject::OnActiveChanged (created or deleted) or ConfigObject::OnVersionChanged (updated) also call UpdateConfigObject().

Event Receiver: ConfigUpdateObjectAPIHandler calls ConfigObjectUtility::CreateObject() in order to create the object if it is not already existing. Afterwards, all modified attributes are applied and in case, original attributes are restored. The object version is set as well, keeping it in sync with the sender.

Permissions

Sender

Client receiver connects:

The sender only syncs config object updates to a client which can access the config object, in ApiListener::SendRuntimeConfigObjects().

In addition to that, the client endpoint's zone is checked whether this zone may access the config object.

Runtime updated object:

Only if the config object belongs to the _api package.

Receiver

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Origin sender endpoint's zone is in a child zone.
• api feature does not accept config
• The received config object type does not exist (this is to prevent failures with older nodes and new object types).

Error handling:

• Log an error if CreateObject fails (only if the object does not already exist)
• Local object version is newer than the received version, object will not be updated.
• Compare modified and original attributes and restore any type of change here.

config::DeleteObject

Location: apilistener-configsync.cpp

Message Body

Key	Value
jsonrpc	2.0
method	config::DeleteObject
params	Dictionary

Params

Key	Type	Description
name	String	Object name.
type	String	Object type name.
version	Number	Object version.

Functions

Event Sender:

ConfigObject::OnActiveChanged (created or deleted) or ConfigObject::OnVersionChanged (updated) call DeleteConfigObject().

Event Receiver: ConfigDeleteObjectAPIHandler

Permissions

Sender

Runtime deleted object:

Only if the config object belongs to the _api package.

Receiver

The receiver will not process messages from not configured endpoints.

Message updates will be dropped when:

• Origin sender endpoint's zone is in a child zone.
• api feature does not accept config
• The received config object type does not exist (this is to prevent failures with older nodes and new object types).
• The object in question was not created at runtime, it does not belong to the _api package.

Error handling:

• Log an error if DeleteObject fails (only if the object does not already exist)

pki::RequestCertificate

Location: jsonrpcconnection-pki.cpp

Message Body

Key	Value
jsonrpc	2.0
method	pki::RequestCertificate
params	Dictionary

Params

Key	Type	Description
ticket	String	Own ticket, or as satellite in CA proxy from local store.
cert_request	String	Certificate request content from local store, optional.

Functions

Event Sender: RequestCertificateHandler Event Receiver: RequestCertificateHandler

Permissions

This is an anonymous request, and the number of anonymous clients can be configured in the api feature.

Only valid certificate request messages are processed, and valid signed certificates won't be signed again.

pki::UpdateCertificate

Location: jsonrpcconnection-pki.cpp

Message Body

Key	Value
jsonrpc	2.0
method	pki::UpdateCertificate
params	Dictionary

Params

Key	Type	Description
status_code	Number	Status code, 0=ok.
cert	String	Signed certificate content.
ca	String	Public CA certificate content.
fingerprint_request	String	Certificate fingerprint from the CSR.

Functions

Event Sender:

• When a client requests a certificate in RequestCertificateHandler and the satellite already has a signed certificate, the pki::UpdateCertificate message is constructed and sent back.
• When the endpoint holding the master's CA private key (and TicketSalt private key) is able to sign the request, the pki::UpdateCertificate message is constructed and sent back.

Event Receiver: UpdateCertificateHandler

Permissions

Message updates are dropped when

• The origin sender is not in a parent zone of the receiver.
• The certificate fingerprint is in an invalid format.

log::SetLogPosition

Location: apilistener.cpp and jsonrpcconnection.cpp

Message Body

Key	Value
jsonrpc	2.0
method	log::SetLogPosition
params	Dictionary

Params

Key	Type	Description
log_position	Timestamp	The endpoint's log position as UNIX timestamp.

Functions

Event Sender:

During log replay to a client endpoint in ApiListener::ReplayLog(), each processed file generates a message which updates the log position timestamp.

ApiListener::ApiTimerHandler() invokes a check to keep all connected endpoints and their log position in sync during replay log.

Event Receiver: SetLogPositionHandler

Permissions

The receiver will not process messages from not configured endpoints.