icinga2/doc/3.04-notifications.md

## Notifications

Notifications on alerts are an integral part of your Icinga 2 monitoring application.
There are many ways of getting a notification to the actual receiver - Email, XMPP,
IRC, Twitter, etc. The default method for executing a notification command are
plugin scripts used for notifications.
These may either be shell commands to invoke a system call to the `mail` binary
or your own script fetching available macro values and doing proper formatting
before sending the notification.
Other mechanism will require writing the notification string into an api processing
it there (for example ticket system integration).

Such notification plugins are available from community users and professionals for
example on the [MonitoringExchange](http://www.monitoringexchange.org) or the
[Icinga Wiki](https://wiki.icinga.org). Or you'll write your own and share it. 
    
A notification requires one or more users (and/or user groups) who will be notified
in case. These users must have all macro attributes defined which will be used in
the `NotificationCommand` on execution, for example `email` as macro dictionary key
is referenced as `$USEREMAIL$`.

The user `icingaadmin` in the example below will get notified only on `WARNING` and
`CRITICAL` states and `problem` and `recovery` notification types.
    
    object User "icingaadmin" {
      display_name = "Icinga 2 Admin",
      enable_notifications = 1,
      notification_state_filter = (StateFilterWarning |
                                   StateFilterCritical),
      notification_type_filter = (NotificationFilterProblem |
                                   NotificationFilterRecovery),  
      macros = {
        "email" = "icinga@localhost",
        "pager" = "+49123456789"
      }
    }
    
> **Note**
>
> If you don't set the `notification_state_filter` and `notification_type_filter`
> configuration attributes for the `User` object, all states and types will be
> notified.

You should choose which information you (and your notified users) are interested in
case of emergency, and also which information does not provide any value to you and
your environment.

> **Note**
>
> The chain of attribute inheritance including the (additive) macro dictionary for
> notifications will allow granular macros for every specific use case, such as
> `$mail$` or `$mobile$` as `User` macros available in `NotificationCommand`.

    Service -> Notification -> Command -> User
       
There are various macros available at runtime execution of the `NotificationCommand`.
The example below may or may not fit your needs.

    object NotificationCommand "mail-service-notification" inherits "plugin-notification-command" {
      command = [ (IcingaSysconfDir + "/icinga2/scripts/mail-notification.sh") ],

      export_macros = [
        "NOTIFICATIONTYPE",
        "SERVICEDESC",
        "HOSTALIAS",
        "HOSTADDRESS",
        "SERVICESTATE",
        "LONGDATETIME",
        "SERVICEOUTPUT",
        "NOTIFICATIONAUTHORNAME",
        "NOTIFICATIONCOMMENT",
        "HOSTDISPLAYNAME",
        "SERVICEDISPLAYNAME",
        "USEREMAIL"
      ]
    }

The command attribute in the `mail-service-notification` command refers to the
shell script installed into `/etc/icinga2/scripts/mail-notification.sh`.
The macros specified in the `export_macros` array are exported as environment
variables and can be used in the notification script.

You can add all shared attributes to a `Notification` template which is inherited
to the defined notifications. That way you'll save duplicated attributes in each
`Notification` object. Attributes can be overridden locally. 
    
    template Notification "generic-notification" {
      notification_interval = 15m,
    
      notification_command = "mail-service-notification",
  
      notification_state_filter = (StateFilterWarning |
                                   StateFilterCritical |
                                   StateFilterUnknown),
      notification_type_filter = (NotificationFilterProblem |
                                  NotificationFilterAcknowledgement |
                                  NotificationFilterRecovery |
                                  NotificationFilterCustom |
                                  NotificationFilterFlappingStart |
                                  NotificationFilterFlappingEnd |
                                  NotificationFilterDowntimeStart |
                                  NotificationFilterDowntimeEnd |
                                  NotificationFilterDowntimeRemoved),
  
      notification_period = "24x7"
    }
    
> **Note**
>
> The `TimePeriod` `24x7` is shipped as example configuration with Icinga 2.

Use the `generic-notification` template for the `mail` notification defined
inline to the host's service `ping4` and assign the `icingaadmin` user. 

    object Host "localhost" {
      services["ping4"] = {
        notifications["mail"] = {
          templates = [ "generic-notification" ],
          notification_command = "mail-notification",
          users = [ "icingaadmin" ],
        }      
      }
    }
    
Notifications can be defined in `Service` templates inherited to the objects.

> **Note**
>
> Instead of assigning users to notifications, you can also add the `user_groups`
> attribute with a list of user groups to the `Notification` object. Icinga 2 will
> resolve all group members and send notifications to all of them.

### Notification Escalations

When a problem notification is sent and a problem still exists after re-notification
you may want to escalate the problem to the next support level. A different approach
is to configure the default notification by email, and escalate the problem via sms
if not already solved.

You can define notification start and end times as additional configuration
attributes making the `Notification` object a so-called `notification escalation`.
Using templates you can share the basic notification attributes such as users or the
`notification_interval` (and override them for the escalation then).

Using the example from above, you can define additional users being escalated for sms
notifications between start and end time.

> **Note**
>
> `notification_state_filter` and `notification_type_filter` configuration attributes
> are not set in this example.

    object User "icinga-oncall-2nd-level" {
      display_name = "Icinga 2nd Level",
      enable_notifications = 1,

      macros = {
        "mobile" = "+49123456781"
      }
    }
    
    object User "icinga-oncall-1st-level" {
      display_name = "Icinga 1st Level",
      enable_notifications = 1,

      macros = {
        "mobile" = "+49123456782"
      }
    }
    
Define an additional `NotificationCommand` for sms notifications.

> **Note**
>
> The example is not complete as there are many different sms providers.
> Please note that sending sms notifications will require an sms provider
> or local hardware with a sim card active.

    object NotificationCommand "sms-notification" {
       command = "$plugindir$/send_sms_notification $mobile$ ..."
    }
    
The two new notification escalations are added onto the host `localhost`
and its service `ping4` using the `generic-notification` template.
The user `icinga-oncall-2nd-level` will get notified by sms (`sms-notification`
command) after `30m` until `1h`.

> **Note**
>
> The `notification_interval` was set to 15m in the `generic-notification`
> template example. Lower that value in your escalations by using a secondary
> template or overriding the attribute directly in the `notifications` array
> position for `escalation-sms-2nd-level`.

If the problem does not get resolved or acknowledged preventing further notifications
the `escalation-sms-1st-level` user will be escalated `1h` after the initial problem was
notified, but only for one hour (`2h` as `end` key for the `times` dictionary). 

    object Host "localhost" {
      services["ping4"] = {
        notifications["mail"] = {
          templates = [ "generic-notification" ],
          notification_command = "mail-notification",
          users = [ "icingaadmin" ],
        },
        notifications["escalation-sms-2nd-level"] = {
          templates = [ "generic-notification" ],
          notification_command = "sms-notification",
          users = [ "icinga-oncall-2nd-level" ],
          
          times = {
            begin = 30m,
            end = 1h
          }
        },
        notifications["escalation-sms-1st-level"] = {
          templates = [ "generic-notification" ],
          notification_command = "sms-notification",
          users = [ "icinga-oncall-1st-level" ],
          
          times = {
            begin = 1h,
            end = 2h
          }
        }        
      }
    }
    
> **Note**
>
> Instead of assigning users to notifications, you can also add the `user_groups`
> attribute with a list of user groups to the `Notification` object. Icinga 2 will
> resolve all group members and send notifications and notification escalations to
> all of them.

### First Notification Delay

Sometimes the problem in question should not be notified when the first notification
happens, but a defined time duration afterwards. In Icinga 2 you can use the `times`
dictionary and set `begin = 15m` as key and value if you want to suppress notifications
in the first 15 minutes. Leave out the `end` key - if not set, Icinga 2 will not check against any
end time for this notification.

    object Host "localhost" {
      services["ping4"] = {
        notifications["mail"] = {
          templates = [ "generic-notification" ],
          notification_command = "mail-notification",
          users = [ "icingaadmin" ],
          
          times = {
            begin = 15m // delay first notification
          }
        }
      }
    }
    
> **Note**
>
> In Icinga 1.x the attribute is called `first_notification_delay`.

### Notification Filters by State and Type

If there are no notification state and type filter attributes defined at the `Notification`
or `User` object Icinga 2 assumes that all states and types are being notified.

> **Note**
>
> In order to notify on problem states, you still need the type filter `NotificationFilterProblem`.

Available state and type filters for notifications are:

    template Notification "generic-notification" {
  
      notification_state_filter = (StateFilterWarning |
                                   StateFilterCritical |
                                   StateFilterUnknown),
      notification_type_filter = (NotificationFilterProblem |
                                  NotificationFilterAcknowledgement |
                                  NotificationFilterRecovery |
                                  NotificationFilterCustom |
                                  NotificationFilterFlappingStart |
                                  NotificationFilterFlappingEnd |
                                  NotificationFilterDowntimeStart |
                                  NotificationFilterDowntimeEnd |
                                  NotificationFilterDowntimeRemoved),
    }

> **Note**
> 
> If you are familiar with Icinga 1.x `notification_options` please note that they have been split
> into type and state, and allow more fine granular filtering for example on downtimes and flapping.
> You can filter for acknowledgements and custom notifications too.