icinga2/doc/3.04-notifications.md

Notifications

Notifications on alerts are an integral part of your Icinga 2 monitoring application. There are many ways of getting a notification to the actual receiver - Email, XMPP, IRC, Twitter, etc. The default method for executing a notification command are plugin scripts used for notifications. These may either be shell commands to invoke a system call to the mail binary or your own script fetching available macro values and doing proper formatting before sending the notification. Other mechanism will require writing the notification string into an api processing it there (for example ticket system integration).

Such notification plugins are available from community users and professionals for example on the MonitoringExchange or the Icinga Wiki. Or you'll write your own and share it.

A notification requires one or more users (and/or user groups) who will be notified in case. These users must have all macro attributes defined which will be used in the NotificationCommand on execution, for example email as macro dictionary key is referenced as $USEREMAIL$.

The user icingaadmin in the example below will get notified only on WARNING and CRITICAL states and problem and recovery notification types.

object User "icingaadmin" {
  display_name = "Icinga 2 Admin",
  enable_notifications = 1,
  notification_state_filter = (StateFilterWarning |
                               StateFilterCritical),
  notification_type_filter = (NotificationFilterProblem |
                               NotificationFilterRecovery),  
  macros = {
    "email" = "icinga@localhost",
    "pager" = "+49123456789"
  }
}

Note

If you don't set the notification_state_filter and notification_type_filter configuration attributes for the User object, all states and types will be notified.

You should choose which information you (and your notified users) are interested in case of emergency, and also which information does not provide any value to you and your environment.

Note

The chain of attribute inheritance including the (additive) macro dictionary for notifications will allow granular macros for every specific use case, such as $mail$ or $mobile$ as User macros available in NotificationCommand.

Service -> Notification -> Command -> User

There are various macros available at runtime execution of the NotificationCommand. The example below may or may not fit your needs.

object NotificationCommand "mail-service-notification" inherits "plugin-notification-command" {
  command = [ (IcingaSysconfDir + "/icinga2/scripts/mail-notification.sh") ],

  export_macros = [
    "NOTIFICATIONTYPE",
    "SERVICEDESC",
    "HOSTALIAS",
    "HOSTADDRESS",
    "SERVICESTATE",
    "LONGDATETIME",
    "SERVICEOUTPUT",
    "NOTIFICATIONAUTHORNAME",
    "NOTIFICATIONCOMMENT",
    "HOSTDISPLAYNAME",
    "SERVICEDISPLAYNAME",
    "USEREMAIL"
  ]
}

The command attribute in the mail-service-notification command refers to the shell script installed into /etc/icinga2/scripts/mail-notification.sh. The macros specified in the export_macros array are exported as environment variables and can be used in the notification script.

You can add all shared attributes to a Notification template which is inherited to the defined notifications. That way you'll save duplicated attributes in each Notification object. Attributes can be overridden locally.

template Notification "generic-notification" {
  notification_interval = 15m,

  notification_command = "mail-service-notification",

  notification_state_filter = (StateFilterWarning |
                               StateFilterCritical |
                               StateFilterUnknown),
  notification_type_filter = (NotificationFilterProblem |
                              NotificationFilterAcknowledgement |
                              NotificationFilterRecovery |
                              NotificationFilterCustom |
                              NotificationFilterFlappingStart |
                              NotificationFilterFlappingEnd |
                              NotificationFilterDowntimeStart |
                              NotificationFilterDowntimeEnd |
                              NotificationFilterDowntimeRemoved),

  notification_period = "24x7"
}

Note

The TimePeriod 24x7 is shipped as example configuration with Icinga 2.

Use the generic-notification template for the mail notification defined inline to the host's service ping4 and assign the icingaadmin user.

object Host "localhost" {
  services["ping4"] = {
    notifications["mail"] = {
      templates = [ "generic-notification" ],
      notification_command = "mail-notification",
      users = [ "icingaadmin" ],
    }      
  }
}

Notifications can be defined in Service templates inherited to the objects.

Note

Instead of assigning users to notifications, you can also add the user_groups attribute with a list of user groups to the Notification object. Icinga 2 will resolve all group members and send notifications to all of them.

Notification Escalations

When a problem notification is sent and a problem still exists after re-notification you may want to escalate the problem to the next support level. A different approach is to configure the default notification by email, and escalate the problem via sms if not already solved.

You can define notification start and end times as additional configuration attributes making the Notification object a so-called notification escalation. Using templates you can share the basic notification attributes such as users or the notification_interval (and override them for the escalation then).

Using the example from above, you can define additional users being escalated for sms notifications between start and end time.

Note

notification_state_filter and notification_type_filter configuration attributes are not set in this example.

object User "icinga-oncall-2nd-level" {
  display_name = "Icinga 2nd Level",
  enable_notifications = 1,

  macros = {
    "mobile" = "+49123456781"
  }
}

object User "icinga-oncall-1st-level" {
  display_name = "Icinga 1st Level",
  enable_notifications = 1,

  macros = {
    "mobile" = "+49123456782"
  }
}

Define an additional NotificationCommand for sms notifications.

Note

The example is not complete as there are many different sms providers. Please note that sending sms notifications will require an sms provider or local hardware with a sim card active.

object NotificationCommand "sms-notification" {
   command = "$plugindir$/send_sms_notification $mobile$ ..."
}

The two new notification escalations are added onto the host localhost and its service ping4 using the generic-notification template. The user icinga-oncall-2nd-level will get notified by sms (sms-notification command) after 30m until 1h.

Note

The notification_interval was set to 15m in the generic-notification template example. Lower that value in your escalations by using a secondary template or overriding the attribute directly in the notifications array position for escalation-sms-2nd-level.

If the problem does not get resolved or acknowledged preventing further notifications the escalation-sms-1st-level user will be escalated 1h after the initial problem was notified, but only for one hour (2h as end key for the times dictionary).

object Host "localhost" {
  services["ping4"] = {
    notifications["mail"] = {
      templates = [ "generic-notification" ],
      notification_command = "mail-notification",
      users = [ "icingaadmin" ],
    },
    notifications["escalation-sms-2nd-level"] = {
      templates = [ "generic-notification" ],
      notification_command = "sms-notification",
      users = [ "icinga-oncall-2nd-level" ],
      
      times = {
        begin = 30m,
        end = 1h
      }
    },
    notifications["escalation-sms-1st-level"] = {
      templates = [ "generic-notification" ],
      notification_command = "sms-notification",
      users = [ "icinga-oncall-1st-level" ],
      
      times = {
        begin = 1h,
        end = 2h
      }
    }        
  }
}

Note

Instead of assigning users to notifications, you can also add the user_groups attribute with a list of user groups to the Notification object. Icinga 2 will resolve all group members and send notifications and notification escalations to all of them.

First Notification Delay

Sometimes the problem in question should not be notified when the first notification happens, but a defined time duration afterwards. In Icinga 2 you can use the times dictionary and set begin = 15m as key and value if you want to suppress notifications in the first 15 minutes. Leave out the end key - if not set, Icinga 2 will not check against any end time for this notification.

object Host "localhost" {
  services["ping4"] = {
    notifications["mail"] = {
      templates = [ "generic-notification" ],
      notification_command = "mail-notification",
      users = [ "icingaadmin" ],
      
      times = {
        begin = 15m // delay first notification
      }
    }
  }
}

Note

In Icinga 1.x the attribute is called first_notification_delay.

Notification Filters by State and Type

If there are no notification state and type filter attributes defined at the Notification or User object Icinga 2 assumes that all states and types are being notified.

Note

In order to notify on problem states, you still need the type filter NotificationFilterProblem.

Available state and type filters for notifications are:

template Notification "generic-notification" {

  notification_state_filter = (StateFilterWarning |
                               StateFilterCritical |
                               StateFilterUnknown),
  notification_type_filter = (NotificationFilterProblem |
                              NotificationFilterAcknowledgement |
                              NotificationFilterRecovery |
                              NotificationFilterCustom |
                              NotificationFilterFlappingStart |
                              NotificationFilterFlappingEnd |
                              NotificationFilterDowntimeStart |
                              NotificationFilterDowntimeEnd |
                              NotificationFilterDowntimeRemoved),
}

Note

If you are familiar with Icinga 1.x notification_options please note that they have been split into type and state, and allow more fine granular filtering for example on downtimes and flapping. You can filter for acknowledgements and custom notifications too.