mirror of https://github.com/Icinga/icinga2.git
300 lines
11 KiB
Markdown
300 lines
11 KiB
Markdown
## Notifications
|
|
|
|
Notifications on alerts are an integral part of your Icinga 2 monitoring application.
|
|
There are many ways of getting a notification to the actual receiver - Email, XMPP,
|
|
IRC, Twitter, etc. The default method for executing a notification command are
|
|
plugin scripts used for notifications.
|
|
These may either be shell commands to invoke a system call to the `mail` binary
|
|
or your own script fetching available macro values and doing proper formatting
|
|
before sending the notification.
|
|
Other mechanism will require writing the notification string into an api processing
|
|
it there (for example ticket system integration).
|
|
|
|
Such notification plugins are available from community users and professionals for
|
|
example on the [MonitoringExchange](http://www.monitoringexchange.org) or the
|
|
[Icinga Wiki](https://wiki.icinga.org). Or you'll write your own and share it.
|
|
|
|
A notification requires one or more users (and/or user groups) who will be notified
|
|
in case. These users must have all macro attributes defined which will be used in
|
|
the `NotificationCommand` on execution, for example `email` as macro dictionary key
|
|
is referenced as `$USEREMAIL$`.
|
|
|
|
The user `icingaadmin` in the example below will get notified only on `WARNING` and
|
|
`CRITICAL` states and `problem` and `recovery` notification types.
|
|
|
|
object User "icingaadmin" {
|
|
display_name = "Icinga 2 Admin",
|
|
enable_notifications = 1,
|
|
notification_state_filter = (StateFilterWarning |
|
|
StateFilterCritical),
|
|
notification_type_filter = (NotificationFilterProblem |
|
|
NotificationFilterRecovery),
|
|
macros = {
|
|
"email" = "icinga@localhost",
|
|
"pager" = "+49123456789"
|
|
}
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> If you don't set the `notification_state_filter` and `notification_type_filter`
|
|
> configuration attributes for the `User` object, all states and types will be
|
|
> notified.
|
|
|
|
You should choose which information you (and your notified users) are interested in
|
|
case of emergency, and also which information does not provide any value to you and
|
|
your environment.
|
|
|
|
> **Note**
|
|
>
|
|
> The chain of attribute inheritance including the (additive) macro dictionary for
|
|
> notifications will allow granular macros for every specific use case, such as
|
|
> `$mail$` or `$mobile$` as `User` macros available in `NotificationCommand`.
|
|
|
|
Service -> Notification -> Command -> User
|
|
|
|
There are various macros available at runtime execution of the `NotificationCommand`.
|
|
The example below may or may not fit your needs.
|
|
|
|
object NotificationCommand "mail-notification" inherits "plugin-notification-command" {
|
|
command = [
|
|
"/usr/bin/printf",
|
|
"\"%b\"",
|
|
{{{\"***** Icinga *****
|
|
|
|
Notification Type: $NOTIFICATIONTYPE$
|
|
|
|
Service: $SERVICEDESC$
|
|
Host: $HOSTALIAS$
|
|
Address: $HOSTADDRESS$
|
|
State: $SERVICESTATE$
|
|
|
|
Date/Time: $LONGDATETIME$
|
|
|
|
Additional Info: $SERVICEOUTPUT$
|
|
|
|
Comment: [$NOTIFICATIONAUTHORNAME$] $NOTIFICATIONCOMMENT$\"}}},
|
|
"/bin/mail",
|
|
"-s",
|
|
"\"$NOTIFICATIONTYPE$ - $HOSTDISPLAYNAME$ - $SERVICEDISPLAYNAME$ is $SERVICESTATE$\"",
|
|
"$USEREMAIL$"
|
|
]
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> Alternatively you can use the `export_macros` attributes to export all required
|
|
> macros into the environment.
|
|
|
|
You can add all shared attributes to a `Notification` template which is inherited
|
|
to the defined notifications. That way you'll save duplicated attributes in each
|
|
`Notification` object. Attributes can be overridden locally.
|
|
|
|
template Notification "generic-notification" {
|
|
notification_interval = 15m,
|
|
|
|
notification_command = "mail-service-notification",
|
|
|
|
notification_state_filter = (StateFilterWarning |
|
|
StateFilterCritical |
|
|
StateFilterUnknown),
|
|
notification_type_filter = (NotificationFilterProblem |
|
|
NotificationFilterAcknowledgement |
|
|
NotificationFilterRecovery |
|
|
NotificationFilterCustom |
|
|
NotificationFilterFlappingStart |
|
|
NotificationFilterFlappingEnd |
|
|
NotificationFilterDowntimeStart |
|
|
NotificationFilterDowntimeEnd |
|
|
NotificationFilterDowntimeRemoved),
|
|
|
|
notification_period = "24x7"
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> The `TimePeriod` `24x7` is shipped as example configuration with Icinga 2.
|
|
|
|
Use the `generic-notification` template for the `mail` notification defined
|
|
inline to the host's service `ping4` and assign the `icingaadmin` user.
|
|
|
|
object Host "localhost" {
|
|
services["ping4"] = {
|
|
notifications["mail"] = {
|
|
templates = [ "generic-notification" ],
|
|
notification_command = "mail-notification",
|
|
users = [ "icingaadmin" ],
|
|
}
|
|
}
|
|
}
|
|
|
|
Notifications can be defined in `Service` templates inherited to the objects.
|
|
|
|
> **Note**
|
|
>
|
|
> Instead of assigning users to notifications, you can also add the `user_groups`
|
|
> attribute with a list of user groups to the `Notification` object. Icinga 2 will
|
|
> resolve all group members and send notifications to all of them.
|
|
|
|
### Notification Escalations
|
|
|
|
When a problem notification is sent and a problem still exists after re-notification
|
|
you may want to escalate the problem to the next support level. A different approach
|
|
is to configure the default notification by email, and escalate the problem via sms
|
|
if not already solved.
|
|
|
|
You can define notification start and end times as additional configuration
|
|
attributes making the `Notification` object a so-called `notification escalation`.
|
|
Using templates you can share the basic notification attributes such as users or the
|
|
`notification_interval` (and override them for the escalation then).
|
|
|
|
Using the example from above, you can define additional users being escalated for sms
|
|
notifications between start and end time.
|
|
|
|
> **Note**
|
|
>
|
|
> `notification_state_filter` and `notification_type_filter` configuration attributes
|
|
> are not set in this example.
|
|
|
|
object User "icinga-oncall-2nd-level" {
|
|
display_name = "Icinga 2nd Level",
|
|
enable_notifications = 1,
|
|
|
|
macros = {
|
|
"mobile" = "+49123456781"
|
|
}
|
|
}
|
|
|
|
object User "icinga-oncall-1st-level" {
|
|
display_name = "Icinga 1st Level",
|
|
enable_notifications = 1,
|
|
|
|
macros = {
|
|
"mobile" = "+49123456782"
|
|
}
|
|
}
|
|
|
|
Define an additional `NotificationCommand` for sms notifications.
|
|
|
|
> **Note**
|
|
>
|
|
> The example is not complete as there are many different sms providers.
|
|
> Please note that sending sms notifications will require an sms provider
|
|
> or local hardware with a sim card active.
|
|
|
|
object NotificationCommand "sms-notification" {
|
|
command = "$plugindir$/send_sms_notification $mobile$ ..."
|
|
}
|
|
|
|
The two new notification escalations are added onto the host `localhost`
|
|
and its service `ping4` using the `generic-notification` template.
|
|
The user `icinga-oncall-2nd-level` will get notified by sms (`sms-notification`
|
|
command) after `30m` until `1h`.
|
|
|
|
> **Note**
|
|
>
|
|
> The `notification_interval` was set to 15m in the `generic-notification`
|
|
> template example. Lower that value in your escalations by using a secondary
|
|
> template or overriding the attribute directly in the `notifications` array
|
|
> position for `escalation-sms-2nd-level`.
|
|
|
|
If the problem does not get resolved or acknowledged preventing further notifications
|
|
the `escalation-sms-1st-level` user will be escalated `1h` after the initial problem was
|
|
notified, but only for one hour (`2h` as `end` key for the `times` dictionary).
|
|
|
|
object Host "localhost" {
|
|
services["ping4"] = {
|
|
notifications["mail"] = {
|
|
templates = [ "generic-notification" ],
|
|
notification_command = "mail-notification",
|
|
users = [ "icingaadmin" ],
|
|
},
|
|
notifications["escalation-sms-2nd-level"] = {
|
|
templates = [ "generic-notification" ],
|
|
notification_command = "sms-notification",
|
|
users = [ "icinga-oncall-2nd-level" ],
|
|
|
|
times = {
|
|
begin = 30m,
|
|
end = 1h
|
|
}
|
|
},
|
|
notifications["escalation-sms-1st-level"] = {
|
|
templates = [ "generic-notification" ],
|
|
notification_command = "sms-notification",
|
|
users = [ "icinga-oncall-1st-level" ],
|
|
|
|
times = {
|
|
begin = 1h,
|
|
end = 2h
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> Instead of assigning users to notifications, you can also add the `user_groups`
|
|
> attribute with a list of user groups to the `Notification` object. Icinga 2 will
|
|
> resolve all group members and send notifications and notification escalations to
|
|
> all of them.
|
|
|
|
### First Notification Delay
|
|
|
|
Sometimes the problem in question should not be notified when the first notification
|
|
happens, but a defined time duration afterwards. In Icinga 2 you can use the `times`
|
|
dictionary and set `begin = 15m` as key and value if you want to suppress notifications
|
|
in the first 15 minutes. Leave out the `end` key - if not set, Icinga 2 will not check against any
|
|
end time for this notification.
|
|
|
|
object Host "localhost" {
|
|
services["ping4"] = {
|
|
notifications["mail"] = {
|
|
templates = [ "generic-notification" ],
|
|
notification_command = "mail-notification",
|
|
users = [ "icingaadmin" ],
|
|
|
|
times = {
|
|
begin = 15m // delay first notification
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> In Icinga 1.x the attribute is called `first_notification_delay`.
|
|
|
|
### Notification Filters by State and Type
|
|
|
|
If there are no notification state and type filter attributes defined at the `Notification`
|
|
or `User` object Icinga 2 assumes that all states and types are being notified.
|
|
|
|
> **Note**
|
|
>
|
|
> In order to notify on problem states, you still need the type filter `NotificationFilterProblem`.
|
|
|
|
Available state and type filters for notifications are:
|
|
|
|
template Notification "generic-notification" {
|
|
|
|
notification_state_filter = (StateFilterWarning |
|
|
StateFilterCritical |
|
|
StateFilterUnknown),
|
|
notification_type_filter = (NotificationFilterProblem |
|
|
NotificationFilterAcknowledgement |
|
|
NotificationFilterRecovery |
|
|
NotificationFilterCustom |
|
|
NotificationFilterFlappingStart |
|
|
NotificationFilterFlappingEnd |
|
|
NotificationFilterDowntimeStart |
|
|
NotificationFilterDowntimeEnd |
|
|
NotificationFilterDowntimeRemoved),
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> If you are familiar with Icinga 1.x `notification_options` please note that they have been split
|
|
> into type and state, and allow more fine granular filtering for example on downtimes and flapping.
|
|
> You can filter for acknowledgements and custom notifications too.
|