icinga2/doc/2.3-monitoring-basics.md

## Monitoring Basics

This part of the Icinga 2 documentation provides an overview of all the basic
monitoring concepts you need to know to run Icinga 2.

### Hosts and Services

Icinga 2 can be used to monitor the availability of hosts and services. Services
can be virtually anything which can be checked in some way:

* Network services (HTTP, SMTP, SNMP, SSH, etc.)
* Printers
* Switches / Routers
* Temperature Sensors
* Other local or network-accessible services

Host objects provide a mechanism to group together services that are running
on the same physical device.

Here is an example of a host object which defines two child services:

    object Host "my-server1" {
      services["ping4"] = {
        check_command = "ping4"
      },

      services["http"] = {
        check_command = "http_ip"
      },

      check = "ping4",

      macros["address"] = "10.0.0.1"
    }

The example host *my-server1* creates two services which belong to this host:
*ping4* and *http*.

It also specifies that the host should inherit its availability state from the
*ping4* service.

> **Note**
>
> In Icinga 1.x hosts had their own check command, check interval and
> notification settings. Instead, in Icinga 2 hosts inherit their state
> from one of its child services. No checks are performed for the host
> itself.

The *address* macro is used by check commands to determine which network
address is associated with the host object.

#### Host States

Hosts inherit their state from the host check service that is specified using
the *check* attribute.

Hosts can be in any of the following states:

  Name        | Description
  ------------|--------------
  UP          | The host is available.
  DOWN        | The host is unavailable.
  UNREACHABLE | At least one of the host's dependencies (e.g. its upstream router) is unavailable causing the host to be unreachable.

#### Service States

Services can be in any of the following states:

  Name        | Description
  ------------|--------------
  OK          | The service is working properly.
  WARNING     | The service is experiencing some problems but is still considered to be in working condition.
  CRITICAL    | The service is in a critical state.
  UNKNOWN     | The check could not determine the service's state.

#### Hard and Soft States

When detecting a problem with a service Icinga re-checks the service a number of
times (based on the *max_check_attempts* and *retry_interval* settings) before sending
notifications. This ensures that no unnecessary notifications are sent for
transient failures. During this time the service is in a *SOFT* state.

After all re-checks have been executed and the service is still in a non-OK
state the service switches to a *HARD* state and notifications are sent.

  Name        | Description
  ------------|--------------
  HARD        | The host/service's state hasn't recently changed.
  SOFT        | The host/service has recently changed state and is being re-checked.

### Check Commands

TODO

### Macros

Macros may be used in command definitions to dynamically change how the command
is executed.

Here is an example of a command definition which uses user-defined macros:

    object CheckCommand "my-ping" inherits "plugin-check-command" {
      command = [
        "$plugindir$/check_ping",
        "-4",
        "-H", "$address$",
        "-w", "$wrta$,$wpl$%",
        "-c", "$crta$,$cpl$%",
        "-p", "$packets$",
        "-t", "$timeout$"
      ],

      macros = {
        wrta = 100,
        wpl = 5,

        crta = 200,
        cpl = 15,

        packets = 5,
        timeout = 0
      }
    }

> **Note**
>
> If you have previously used Icinga 1.x you may already be familiar with
> user and argument macros (e.g., USER1 or ARG1). Unlike in Icinga 1.x macros
> may have arbitrary names and arguments are no longer specified in the
> check_command setting.

Macro names must be enclosed in two *$* signs, e.g. *$plugindir$*. When
executing commands Icinga 2 checks the following objects in this
order to look up macros:

1. User object (only for notifications)
2. Service object
3. Host object
4. Command object
5. Global macros in the IcingaMacros variable

This execution order allows you to define default values for macros in your
command objects. The *my-ping* command shown above uses this to set default
values for some of the latency thresholds and timeouts.

When using the *my-ping* command you can override all or some of the macros
in the service definition like this:

    object Host "my-server1" {
      services["ping"] = {
        check_command = "my-ping",

        macros["packets"] = 10 // Overrides the default value of 5 given in the command
      },

      macros["address"] = "10.0.0.1"
    }

If a macro isn't defined anywhere an empty value is used and a warning is
emitted to the Icinga 2 log.

> **Note**
>
> Macros in capital letters (e.g. HOSTNAME) are reserved for use by Icinga 2
> and should not be overwritten by users.

By convention every host should have an *address* macro. Hosts
which have an IPv6 address should also have an *address6* macro.

The *plugindir* macro should be set to the path of your check plugins. The
*/etc/icinga2/conf.d/macros.conf* file is usually used to define global macros
including this one.

#### Host Macros

The following host macros are available in all commands that are executed for
hosts or services:

  Name                   | Description
  -----------------------|--------------
  HOSTNAME               | The name of the host object.
  HOSTDISPLAYNAME        | The value of the display_name attribute.
  HOSTALIAS              | This is an alias for the *HOSTDISPLAYNAME* macro.
  HOSTSTATE              | The host's current state. Can be one of UNREACHABLE, UP and DOWN.
  HOSTSTATEID            | The host's current state. Can be one of 0 (up), 1 (down) and 2 (unreachable).
  HOSTSTATETYPE          | The host's current state type. Can be one of SOFT and HARD.
  HOSTATTEMPT            | The current check attempt number.
  MAXHOSTATTEMPT         | The maximum number of checks which are executed before changing to a hard state.
  LASTHOSTSTATE          | The host's previous state. Can be one of UNREACHABLE, UP and DOWN.
  LASTHOSTSTATEID        | The host's previous state. Can be one of 0 (up), 1 (down) and 2 (unreachable).
  LASTHOSTSTATETYPE      | The host's previous state type. Can be one of SOFT and HARD.
  HOSTLATENCY            | The host's check latency.
  HOSTEXECUTIONTIME      | The host's check execution time.
  HOSTOUTPUT             | The last check's output.
  HOSTPERFDATA           | The last check's performance data.
  LASTHOSTCHECK          | The timestamp when the last check was executed.
  HOSTADDRESS            | This is an alias for the *address* macro. If the *address* macro is not defined the host object's name is used instead.
  HOSTADDRESS6           | This is an alias for the *address6* macro. If the *address* macro is not defined the host object's name is used instead.

#### Service Macros

The following service macros are available in all commands that are executed for
services:

  Name                   | Description
  -----------------------|--------------
  SERVICEDESC            | The short name of the service object.
  SERVICEDISPLAYNAME     | The value of the display_name attribute.
  SERVICECHECKCOMMAND    | This is an alias for the *SERVICEDISPLAYNAME* macro.
  SERVICESTATE           | The service's current state. Can be one of OK, WARNING, CRITICAL, UNCHECKABLE and UNKNOWN.
  SERVICESTATEID         | The service's current state. Can be one of 0 (ok), 1 (warning), 2 (critical), 3 (unknown) and 4 (uncheckable).
  SERVICESTATETYPE       | The service's current state type. Can be one of SOFT and HARD.
  SERVICEATTEMPT         | The current check attempt number.
  MAXSERVICEATTEMPT      | The maximum number of checks which are executed before changing to a hard state.
  LASTSERVICESTATE       | The service's previous state. Can be one of OK, WARNING, CRITICAL, UNCHECKABLE and UNKNOWN.
  LASTSERVICESTATEID     | The service's previous state. Can be one of 0 (ok), 1 (warning), 2 (critical), 3 (unknown) and 4 (uncheckable).
  LASTSERVICESTATETYPE   | The service's previous state type. Can be one of SOFT and HARD.
  LASTSERVICESTATECHANGE | The last state change's timestamp.
  SERVICELATENCY         | The service's check latency.
  SERVICEEXECUTIONTIME   | The service's check execution time.
  SERVICEOUTPUT          | The last check's output.
  SERVICEPERFDATA        | The last check's performance data.
  LASTSERVICECHECK       | The timestamp when the last check was executed.

#### User Macros

The following service macros are available in all commands that are executed for
users:

  Name                   | Description
  -----------------------|--------------
  CONTACTNAME            | The name of the user object.
  CONTACTALIAS           | The value of the display_name attribute.
  CONTACTEMAIL           | This is an alias for the *email* macro.
  CONTACTPAGER           | This is an alias for the *pager* macro.

#### Global Macros

The following macros are available in all commands:

  Name                   | Description
  -----------------------|--------------
  TIMET                  | Current UNIX timestamp.
  LONGDATETIME           | Current date and time including timezone information.
  SHORTDATETIME          | Current date and time.
  DATE                   | Current date.
  TIME                   | Current time including timezone information.

### Using Templates

Templates may be used to apply a set of similar settings to more than one
object.

For example, rather than manually creating a *ping* service object for each of
your hosts you can use templates to avoid having to copy & paste parts of your
config:

    template Host "linux-server" {
      services["ping"] = {
        check_command = "ping4"
      },

      check = "ping4"
    }

    object Host "my-server1" inherits "linux-server" {
      macros["address"] = "10.0.0.1"
    }

    object Host "my-server2" inherits "linux-server" {
      macros["address"] = "10.0.0.2"
    }

In this example both *my-server1* and *my-server2* each get their own ping
service check.

Objects as well as templates themselves can inherit from an arbitrary number of
templates. Attributes inherited from a template can be overridden in the
object if necessary.

### Groups

TODO

### Host/Service Dependencies

TODO

### Notifications

TODO