mirror of https://github.com/Icinga/icinga2.git
291 lines
11 KiB
Markdown
291 lines
11 KiB
Markdown
## <a id="notifications"></a> Notifications
|
|
|
|
Notifications on alerts are an integral part of your Icinga 2 monitoring application.
|
|
There are many ways of getting a notification to the actual receiver - Email, XMPP,
|
|
IRC, Twitter, etc. The default method for executing a notification command are
|
|
plugin scripts used for notifications.
|
|
These may either be shell commands to invoke a system call to the `mail` binary
|
|
or your own script fetching available custom attribute values and doing proper formatting
|
|
before sending the notification.
|
|
Other mechanism will require writing the notification string into an api processing
|
|
it there (for example ticket system integration).
|
|
|
|
Such notification plugins are available from community users and professionals for
|
|
example on the [MonitoringExchange](http://www.monitoringexchange.org) or the
|
|
[Icinga Wiki](https://wiki.icinga.org). Or you'll write your own and share it.
|
|
|
|
A notification requires one or more users (and/or user groups) who will be notified
|
|
in case. These users must have all custom attributes defined which will be used in
|
|
the `NotificationCommand` on execution.
|
|
|
|
TODO
|
|
|
|
The user `icingaadmin` in the example below will get notified only on `WARNING` and
|
|
`CRITICAL` states and `problem` and `recovery` notification types.
|
|
|
|
object User "icingaadmin" {
|
|
display_name = "Icinga 2 Admin"
|
|
enable_notifications = 1
|
|
notification_state_filter = (StateFilterOK |
|
|
StateFilterWarning |
|
|
StateFilterCritical)
|
|
notification_type_filter = (NotificationFilterProblem |
|
|
NotificationFilterRecovery)
|
|
vars.email = "icinga@localhost"
|
|
vars.pager = "+49123456789"
|
|
}
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> If you don't set the `notification_state_filter` and `notification_type_filter`
|
|
> configuration attributes for the `User` object, all states and types will be
|
|
> notified.
|
|
> Recovery notifications require the state filter `StateFilterOK`.
|
|
|
|
You should choose which information you (and your notified users) are interested in
|
|
case of emergency, and also which information does not provide any value to you and
|
|
your environment.
|
|
|
|
> **Note**
|
|
>
|
|
> The chain of attribute inheritance including the (additive) vars dictionary for
|
|
> notifications will allow granular custom attributes for every specific use case.
|
|
|
|
|
|
There are various custom attributes available at runtime execution of the
|
|
`NotificationCommand`. The example below may or may not fit your needs.
|
|
|
|
object NotificationCommand "mail-service-notification" {
|
|
import "plugin-notification-command"
|
|
|
|
command = [ IcingaSysconfDir + "/icinga2/scripts/mail-notification.sh" ]
|
|
|
|
env = {
|
|
"NOTIFICATIONTYPE" = "$notification.type$"
|
|
"SERVICEDESC" = "$service.description$"
|
|
"HOSTALIAS" = "$host.displayname$",
|
|
"HOSTADDRESS" = "$host.vars.address$",
|
|
"SERVICESTATE" = "$service.state$",
|
|
"LONGDATETIME" = "$icinga.longdatetime$",
|
|
"SERVICEOUTPUT" = "$service.output$",
|
|
"NOTIFICATIONAUTHORNAME" = "$notification.author$",
|
|
"NOTIFICATIONCOMMENT" = "$notification.comment$",
|
|
"HOSTDISPLAYNAME" = "$host.displayname$",
|
|
"SERVICEDISPLAYNAME" = "$service.displayname$",
|
|
"USEREMAIL" = "$user.vars.email$"
|
|
}
|
|
}
|
|
|
|
The command attribute in the `mail-service-notification` command refers to the
|
|
shell script installed into `/etc/icinga2/scripts/mail-notification.sh`.
|
|
The macros specified in the `export_macros` array are exported as environment
|
|
variables and can be used in the notification script.
|
|
|
|
You can add all shared attributes to a `Notification` template which is inherited
|
|
to the defined notifications. That way you'll save duplicated attributes in each
|
|
`Notification` object. Attributes can be overridden locally.
|
|
|
|
TODO
|
|
|
|
template Notification "generic-notification" {
|
|
notification_interval = 15m
|
|
|
|
notification_command = "mail-service-notification"
|
|
|
|
notification_state_filter = (StateFilterWarning |
|
|
StateFilterCritical |
|
|
StateFilterUnknown)
|
|
notification_type_filter = (NotificationFilterProblem |
|
|
NotificationFilterAcknowledgement |
|
|
NotificationFilterRecovery |
|
|
NotificationFilterCustom |
|
|
NotificationFilterFlappingStart |
|
|
NotificationFilterFlappingEnd |
|
|
NotificationFilterDowntimeStart |
|
|
NotificationFilterDowntimeEnd |
|
|
NotificationFilterDowntimeRemoved)
|
|
|
|
notification_period = "24x7"
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> The `TimePeriod` `24x7` is shipped as example configuration with Icinga 2.
|
|
|
|
Use the `apply` keyword to create `Notification` objects for your services:
|
|
|
|
apply Notification "mail" to Service {
|
|
import "generic-notification"
|
|
|
|
notification_command = "mail-notification"
|
|
users = [ "icingaadmin" ]
|
|
|
|
assign where service.name == "ping4"
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> Instead of assigning users to notifications, you can also add the `user_groups`
|
|
> attribute with a list of user groups to the `Notification` object. Icinga 2 will
|
|
> resolve all group members and send notifications to all of them.
|
|
|
|
### <a id="notification-escalations"></a> Notification Escalations
|
|
|
|
When a problem notification is sent and a problem still exists after re-notification
|
|
you may want to escalate the problem to the next support level. A different approach
|
|
is to configure the default notification by email, and escalate the problem via sms
|
|
if not already solved.
|
|
|
|
You can define notification start and end times as additional configuration
|
|
attributes making the `Notification` object a so-called `notification escalation`.
|
|
Using templates you can share the basic notification attributes such as users or the
|
|
`notification_interval` (and override them for the escalation then).
|
|
|
|
Using the example from above, you can define additional users being escalated for sms
|
|
notifications between start and end time.
|
|
|
|
> **Note**
|
|
>
|
|
> `notification_state_filter` and `notification_type_filter` configuration attributes
|
|
> are not set in this example.
|
|
|
|
object User "icinga-oncall-2nd-level" {
|
|
display_name = "Icinga 2nd Level"
|
|
enable_notifications = true
|
|
|
|
vars.mobile = "+49123456781"
|
|
}
|
|
|
|
object User "icinga-oncall-1st-level" {
|
|
display_name = "Icinga 1st Level"
|
|
enable_notifications = true
|
|
|
|
vars.mobile = "+49123456782"
|
|
}
|
|
}
|
|
|
|
Define an additional `NotificationCommand` for sms notifications.
|
|
|
|
> **Note**
|
|
>
|
|
> The example is not complete as there are many different sms providers.
|
|
> Please note that sending sms notifications will require an sms provider
|
|
> or local hardware with a sim card active.
|
|
|
|
object NotificationCommand "sms-notification" {
|
|
command = PluginDir + "/send_sms_notification $mobile$ ..."
|
|
}
|
|
|
|
The two new notification escalations are added onto the host `localhost`
|
|
and its service `ping4` using the `generic-notification` template.
|
|
The user `icinga-oncall-2nd-level` will get notified by sms (`sms-notification`
|
|
command) after `30m` until `1h`.
|
|
|
|
> **Note**
|
|
>
|
|
> The `notification_interval` was set to 15m in the `generic-notification`
|
|
> template example. Lower that value in your escalations by using a secondary
|
|
> template or overriding the attribute directly in the `notifications` array
|
|
> position for `escalation-sms-2nd-level`.
|
|
|
|
If the problem does not get resolved or acknowledged preventing further notifications
|
|
the `escalation-sms-1st-level` user will be escalated `1h` after the initial problem was
|
|
notified, but only for one hour (`2h` as `end` key for the `times` dictionary).
|
|
|
|
apply Notification "mail" to Service {
|
|
import "generic-notification"
|
|
notification_command = "mail-notification"
|
|
users = [ "icingaadmin" ]
|
|
|
|
assign where service.name == "ping4"
|
|
}
|
|
|
|
apply Notification "escalation-sms-2nd-level" to Service {
|
|
import "generic-notification"
|
|
notification_command = "sms-notification"
|
|
users = [ "icinga-oncall-2nd-level" ]
|
|
|
|
times = {
|
|
begin = 30m
|
|
end = 1h
|
|
}
|
|
|
|
assign where service.name == "ping4"
|
|
}
|
|
|
|
apply Notification "escalation-sms-1st-level" to Service {
|
|
import "generic-notification"
|
|
notification_command = "sms-notification"
|
|
users = [ "icinga-oncall-1st-level" ]
|
|
|
|
times = {
|
|
begin = 1h
|
|
end = 2h
|
|
}
|
|
|
|
assign where service.name == "ping4"
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> Instead of assigning users to notifications, you can also add the `user_groups`
|
|
> attribute with a list of user groups to the `Notification` object. Icinga 2 will
|
|
> resolve all group members and send notifications and notification escalations to
|
|
> all of them.
|
|
|
|
### <a id="first-notification-delay"></a> First Notification Delay
|
|
|
|
Sometimes the problem in question should not be notified when the first notification
|
|
happens, but a defined time duration afterwards. In Icinga 2 you can use the `times`
|
|
dictionary and set `begin = 15m` as key and value if you want to suppress notifications
|
|
in the first 15 minutes. Leave out the `end` key - if not set, Icinga 2 will not check against any
|
|
end time for this notification.
|
|
|
|
apply Notification "mail" to Service {
|
|
import "generic-notification"
|
|
notification_command = "mail-notification"
|
|
users = [ "icingaadmin" ]
|
|
|
|
times.begin = 15m // delay first notification
|
|
|
|
assign where service.name == "ping4"
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> In Icinga 1.x the attribute is called `first_notification_delay`.
|
|
|
|
### <a id="notification-filters-state-type"></a> Notification Filters by State and Type
|
|
|
|
If there are no notification state and type filter attributes defined at the `Notification`
|
|
or `User` object Icinga 2 assumes that all states and types are being notified.
|
|
|
|
> **Note**
|
|
>
|
|
> In order to notify on problem states, you still need the type filter `NotificationFilterProblem`.
|
|
|
|
Available state and type filters for notifications are:
|
|
|
|
template Notification "generic-notification" {
|
|
|
|
notification_state_filter = (StateFilterWarning |
|
|
StateFilterCritical |
|
|
StateFilterUnknown)
|
|
notification_type_filter = (NotificationFilterProblem |
|
|
NotificationFilterAcknowledgement |
|
|
NotificationFilterRecovery |
|
|
NotificationFilterCustom |
|
|
NotificationFilterFlappingStart |
|
|
NotificationFilterFlappingEnd |
|
|
NotificationFilterDowntimeStart |
|
|
NotificationFilterDowntimeEnd |
|
|
NotificationFilterDowntimeRemoved)
|
|
}
|
|
|
|
> **Note**
|
|
>
|
|
> If you are familiar with Icinga 1.x `notification_options` please note that they have been split
|
|
> into type and state, and allow more fine granular filtering for example on downtimes and flapping.
|
|
> You can filter for acknowledgements and custom notifications too.
|