icinga2/docs/icinga2-intro.md

Icinga 2 is a network monitoring application that tries to improve upon
the success of Icinga 1.x while fixing some of its shortcomings. A few
frequently encountered issues are:

-   Scalability problems in large monitoring setups

-   Difficult configuration with dozens of "magic" tweaks and several
    ways of defining services

-   Code quality and the resulting inability to implement changes
    without breaking add-ons

-   Limited access to the runtime state of Icinga (e.g. for querying a
    service’s state or for dynamically creating new services)

Fixing these issues would involve major breaking changes to the Icinga
1.x core and configuration syntax. Icinga users would likely experience
plenty of problems with the Icinga versions introducing these changes.
Many of these changes would likely break add-ons which rely on the NEB
API and other core internals.

From a developer standpoint this may be justifiable in order to get to a
better end-product. However, for (business) users spending time on
getting familiar with these changes for each new version may become
quite frustrating and may easily cause users to lose their confidence in
Icinga.

Nagios™ 4 is currently following this approach and it remains to be seen
how this fares with its users.

Instead the Icinga project will maintain two active development
branches. There will be one branch for Icinga 1.x which focuses on
improving the existing Icinga 1.x code base - just like it has been done
so far.

Independently from Icinga 1.x development on Icinga 2 will happen in a
separate branch and some of the long-term design goals will be outlined
in this document. Status updates for Icinga 2 will be posted on the
project website (www.icinga.org) as they become available.

Code Quality
============

Icinga 2 will not be using any code from the Icinga 1.x branch due to
the rampant code quality issues with the existing code base. However, an
important property of the Icinga development process has always been to
rely on proven technologies and Icinga 2 will be no exception.

A lot of effort has gone into designing a maintainable architecture for
Icinga 2 and making sure that algorithmic choices are in alignment with
our scalability goals for Icinga 2.

There are plans to implement unit tests for most Icinga 2 features in
order to make sure that changes to the code base do not break things
that were known to work before.

Language Choice
===============

Icinga 1.x is written in C and while in general C has quite a number of
advantages (e.g. performance and relatively easy portability to other
\*NIX- based platforms) some of its disadvantages show in the context of
a project that is as large as Icinga.

With a complex software project like Icinga an object-oriented design
helps tremendously with keeping things modular and making changes to the
existing code easier.

While it is true that you can write object-oriented software in C (the
Linux kernel is one of the best examples of how to do that) a truly
object-oriented language makes the programmers' life just a little bit
easier.

For Icinga 2 we have chosen C++ as the main language. This decision was
influenced by a number of criteria including performance, support on
different platforms and general user acceptability.

In general there is nothing wrong with other languages like Java, C\# or
Python; however - even when ignoring technical problems for just a
moment - in a community as conservative as the monitoring community
these languages seem out of place.

Knowing that users will likely want to run Icinga 2 on older systems
(which are still fully vendor-supported even for years to come) we will
make every effort to ensure that Icinga 2 can be built and run on
commonly used operating systems and refrain from using new and exotic
features like C++11.

Unlike Icinga 1.x there will be Windows support for Icinga 2. Some of
the compatibility features (e.g. the command pipe) which rely on \*NIX
features may not be supported on Windows but all new features will be
designed in such a way as to support \*NIX as well as Windows.

Configuration
=============

Icinga 1.x has a configuration format that is fully backwards-compatible
to the Nagios™ configuration format. This has the advantage of allowing
users to easily upgrade their existing Nagios™ installations as well as
downgrading if they choose to do so (even though this is generally not
the case).

The Nagios™ configuration format has evolved organically over time and
for the most part it does what it’s supposed to do. However this
evolutionary process has brought with it a number of problems that make
it difficult for new users to understand the full breadth of available
options and ways of setting up their monitoring environment.

Experience with other configuration formats like the one used by Puppet
has shown that it is often better to have a single "right" way of doing
things rather than having multiple ways like Nagios™ does (e.g. defining
host/service dependencies and parent/child relationships for hosts).

Icinga 2 tries to fix those issues by introducing a new object-based
configuration format that is heavily based on templates and supports
user-friendly features like freely definable macros.

External Interfaces
===================

While Icinga 1.x has easily accessible interfaces to its internal state
(e.g. status.dat, objects.cache and the command pipe) there is no
standards-based way of getting that information.

For example, using Icinga’s status information in a custom script
generally involves writing a parser for the status.dat format and there
are literally dozens of Icinga-specific status.dat parsers out there.

While Icinga 2 will support these legacy interfaces in order to make
migration easier and allowing users to use the existing CGIs and
whatever other scripts they may have Icinga 2 will focus on providing a
unified interface to Icinga’s state and providing similar functionality
to that provided by the command pipe in Icinga 1.x. The exact details
for such an interface are yet to be determined but this will likely be
an RPC interface based on one of the commonly used web-based remoting
technologies.

Icinga 1.x exports historical data using the IDO database interface
(Icinga Data Output). Icinga 2 will support IDO in a
backwards-compatible fashion in order to support icinga-web.
Additionally there will be a newly-designed backend for historical data
which can be queried using the built-in API when available. Effort will
be put into making this new data source more efficient for use with SLA
reporting.

Icinga 2 will also feature dynamic reconfiguration using the API which
means users can create, delete and update any configuration object (e.g.
hosts and services) on-the-fly. Based on the API there are plans to
implement a command-line configuration tool similar to what Pacemaker
has with "crm". Later on this API may also be used to implement
auto-discovery for new services.

The RPC interface may also be used to receive events in real-time, e.g.
when service checks are being executed or when a service’s state
changes. Some possible uses of this interface would be to export
performance data for services (RRD, graphite, etc.) or general log
information (logstash, graylog2, etc.).

Checks
======

In Icinga 2 services are the only checkable objects. Hosts only have a
calculated state and no check are ever run for them.

In order to maintain compatibility with the hundreds of existing check
plugins for Icinga 1.x there will be support for Nagios™-style checks.
The check interface however will be modular so that support for other
kinds of checks can be implemented later on (e.g. built-in checks for
commonly used services like PING, HTTP, etc. in order to avoid spawning
a process for each check).

Based on the availability of remote Icinga 2 instances the core can
delegate execution of service checks to them in order to support
large-scale distributed setups with a minimal amount of maintenance.
Services can be assigned to specific check instances using configuration
settings.

Notifications
=============

Event handlers and notifications will be supported similar to Icinga
1.x. Thanks to the dynamic configuration it is possible to easily adjust
the notification settings at runtime (e.g. in order to implement on-call
rotation).

Scalability
===========

Icinga 1.x has some serious scalability issues which explains why there
are several add-ons which try to improve the core’s check performance.
One of these add-ons is mod\_gearman which can be used to distribute
checks to multiple workers running on remote systems.

A problem that remains is the performance of the core when processing
check results. Scaling Icinga 1.x beyond 25.000 services proves to be a
challenging problem and usually involves setting up a cascade of Icinga
1.x instances and dividing the service checks between those instances.
This significantly increases the maintenance overhead when updating the
configuration for such a setup.

Icinga 2 natively supports setting up multiple Icinga 2 instances in a
cluster to distribute work between those instances. Independent tasks
(e.g. performing service checks, sending notifications, updating the
history database, etc.) are implemented as components which can be
loaded for each instance. Configuration as well as program state is
automatically replicated between instances.

In order to support using Icinga 2 in a partially trusted environment
SSL is used for all network communication between individual instances.
Objects (like hosts and services) can be grouped into security domains
for which permissions can be specified on a per-instance basis (so e.g.
you can have a separate API or checker instance for a specific domain).

Agent-based Checks
==================

Traditionally most service checks have been performed actively, meaning
that check plugins are executed on the same server that is also running
Icinga. This works great for checking most network-based services, e.g.
PING and HTTP. However, there are a number of services which cannot be
checked remotely either because they are not network-based or because
firewall settings or network policies ("no unencrypted traffic")
disallow accessing these services from the network where Icinga is
running.

To solve this problem two add-ons have emerged, namely NRPE and NSCA.
NRPE can be thought of as a light-weight remote shell which allows the
execution of a restricted set of commands while supporting some
Nagios™-specific concepts like command timeouts. However unlike with the
design of commonly used protocols like SSH security in NRPE is merely an
afterthought.

In most monitoring setups all NRPE agents share the same secret key
which is embedded into the NRPE binary at compile time. This means that
users can extract this secret key from their NRPE agent binary and use
it to query sensitive monitoring information from other systems running
the same NRPE binary. NSCA has similar problems.

Based on Icinga 2’s code for check execution there will be an agent
which can be used on \*NIX as well as on Windows platforms. The agent
will be using the same configuration format like Icinga 2 itself and
will support SSL and IPv4/IPv6 to communicate with Icinga 2.

Business Processes
==================

In most cases users don’t care about the availability of individual
services but rather the aggregated state of multiple related services.
For example one might have a database cluster that is used for a web
shop. For an end-user the shop is available as long as at least one of
the database servers is working.

Icinga 1.x does not have any support for business processes out of the
box. There are several add-ons which implement business process support
for Icinga, however none of those are well-integrated into Icinga.

Icinga 2 will have native support for business processes which are built
right into the core and can be configured in a similar manner to
Nagios™-style checks. Users can define their own services based on
business rules which can be used as dependencies for other hosts or
services.

Logging
=======

Icinga 2 supports file-based logged as well as syslog (on \*NIX) and
event log (on Windows). Additionally Icinga 2 supports remote logging to
a central Icinga 2 instance.