tyler.durden
/
icinga2
mirror of https://github.com/Icinga/icinga2.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #5136 from dgoetz/feature/improve-eventhandler-docs 

Enhance event command docs with more examples on remote clients
This commit is contained in:
Michael Friedrich 2017-05-09 17:23:03 +02:00 committed by GitHub
parent 452f8a1f0d 8e83a920cf
commit f3e9c86f2c
2 changed files with 278 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

294
doc/3-monitoring-basics.md
View File

@ -1631,33 +1631,292 @@ check execution if one of these conditions matches:
[EventCommand](9-object-types.md#objecttype-eventcommand) objects are referenced by [EventCommand](9-object-types.md#objecttype-eventcommand) objects are referenced by
[Host](9-object-types.md#objecttype-host) and [Service](9-object-types.md#objecttype-service) objects [Host](9-object-types.md#objecttype-host) and [Service](9-object-types.md#objecttype-service) objects
using the `event_command` attribute. with the `event_command` attribute.
Therefore the `EventCommand` object should define a command line Therefore the `EventCommand` object should define a command line
evaluating the current service state and other service runtime attributes evaluating the current service state and other service runtime attributes
available through runtime vars. Runtime macros such as `$service.state_type$` available through runtime variables. Runtime macros such as `$service.state_type$`
and `$service.state$` will be processed by Icinga 2 helping on fine-granular and `$service.state$` will be processed by Icinga 2 and help with fine-granular
events being triggered. triggered events
If you are using a client as [command endpoint](6-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) If the host/service is located on a client as [command endpoint](6-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)
the event command will be executed on the client itself (similar to the check the event command will be executed on the client itself (similar to the check
command). command).
Common use case scenarios are a failing HTTP check requiring an immediate Common use case scenarios are a failing HTTP check which requires an immediate
restart via event command, or if an application is locked and requires restart via event command. Another example would be an application that is not
a restart upon detection. responding and therefore requires a restart. You can also use event handlers
to forward more details on state changes and events than the typical notification
alerts provide.
#### <a id="event-command-restart-service-daemon"></a> Use Event Commands to Restart Service Daemon #### <a id="event-command-send-information-from-master"></a> Use Event Commands to Send Information from the Master
The following example will trigger a restart of the `httpd` daemon This example sends a web request from the master node to an external tool
via ssh when the `http` service check fails. If the service state is for every event triggered on a `businessprocess` service.
`OK`, it will not trigger any event action.
Define an [EventCommand](9-object-types.md#objecttype-eventcommand)
object `send_to_businesstool` which sends state changes to the external tool.
object EventCommand "send_to_businesstool" {
command = [
"/usr/bin/curl",
"-s",
"-X PUT"
]
arguments = {
"-H" = {
value ="$businesstool_url$"
skip_key = true
}
"-d" = "$businesstool_message$"
}
vars.businesstool_url = "http://localhost:8080/businesstool"
vars.businesstool_message = "$host.name$ $service.name$ $service.state$ $service.state_type$ $service.check_attempt$"
}
Set the `event_command` attribute to `send_to_businesstool` on the Service.
object Service "businessprocess" {
host_name = "businessprocess"
check_command = "icingacli-businessprocess"
vars.icingacli_businessprocess_process = "icinga"
vars.icingacli_businessprocess_config = "training"
event_command = "send_to_businesstool"
}
In order to test this scenario you can run:
nc -l 8080
This allows to catch the web request. You can also enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output)
and search for the event command execution log message.
tail -f /var/log/icinga2/debug.log | grep EventCommand
Feed in a check result via REST API action [process-check-result](12-icinga2-api.md#icinga2-api-actions-process-check-result)
or via Icinga Web 2.
Expected Result:
# nc -l 8080
PUT /businesstool HTTP/1.1
User-Agent: curl/7.29.0
Host: localhost:8080
Accept: */*
Content-Length: 47
Content-Type: application/x-www-form-urlencoded
businessprocess businessprocess CRITICAL SOFT 1
#### <a id="event-command-restart-service-daemon-command-endpoint-linux"></a> Use Event Commands to Restart Service Daemon via Command Endpoint on Linux
This example triggers a restart of the `httpd` service on the local system
when the `procs` service check executed via Command Endpoint fails. It only
triggers if the service state is `Critical` and attempts to restart the
service before a notification is sent.
Requirements: Requirements:
* ssh connection * Icinga 2 as client on the remote node
* icinga user with sudo permissions to the httpd daemon
Example on CentOS 7:
# visudo
icinga ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart httpd
Note: Distributions might use a different name. On Debian/Ubuntu the service is called `apache2`.
Define an [EventCommand](9-object-types.md#objecttype-eventcommand) object `restart_service`
which allows to trigger local service restarts. Put it into a [global zone](6-distributed-monitoring.md#distributed-monitoring-global-zone-config-sync)
to sync its configuration to all clients.
[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/global-templates/eventcommands.conf
object EventCommand "restart_service" {
command = [ PluginDir + "/restart_service" ]
arguments = {
"-s" = "$service.state$"
"-t" = "$service.state_type$"
"-a" = "$service.check_attempt$"
"-S" = "$restart_service$"
}
vars.restart_service = "$procs_command$"
}
This event command triggers the following script which restarts the service.
The script only is executed if the service state is `CRITICAL`. Warning and Unknown states
are ignored as they indicate not an immediate failure.
[root@icinga2-client1.localdomain /]# vim /usr/lib64/nagios/plugins/restart_service
#!/bin/bash
while getopts "s:t:a:S:" opt; do
case $opt in
s)
servicestate=$OPTARG
;;
t)
servicestatetype=$OPTARG
;;
a)
serviceattempt=$OPTARG
;;
S)
service=$OPTARG
;;
esac
done
if ( [ -z $servicestate ] || [ -z $servicestatetype ] || [ -z $serviceattempt ] || [ -z $service ] ); then
echo "USAGE: $0 -s servicestate -z servicestatetype -a serviceattempt -S service"
exit 3;
else
# Only restart on the third attempt of a critical event
if ( [ $servicestate == "CRITICAL" ] && [ $servicestatetype == "SOFT" ] && [ $serviceattempt -eq 3 ] ); then
sudo /usr/bin/systemctl restart $service
fi
fi
[root@icinga2-client1.localdomain /]# chmod +x /usr/lib64/nagios/plugins/restart_service
Add a service on the master node which is executed via command endpoint on the client.
Set the `event_command` attribute to `restart_service`, the name of the previously defined
EventCommand object.
[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/icinga2-client1.localdomain.conf
object Service "Process httpd" {
check_command = "procs"
event_command = "restart_service"
max_check_attempts = 4
host_name = "icinga2-client1.localdomain"
command_endpoint = "icinga2-client1.localdomain"
vars.procs_command = "httpd"
vars.procs_warning = "1:10"
vars.procs_critical = "1:"
}
In order to test this configuration just stop the `httpd` on the remote host `icinga2-client1.localdomain`.
[root@icinga2-client1.localdomain /]# systemctl stop httpd
You can enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) and search for the
executed command line.
[root@icinga2-client1.localdomain /]# tail -f /var/log/icinga2/debug.log | grep restart_service
#### <a id="event-command-restart-service-daemon-command-endpoint-windows"></a> Use Event Commands to Restart Service Daemon via Command Endpoint on Windows
This example triggers a restart of the `httpd` service on the remote system
when the `service-windows` service check executed via Command Endpoint fails.
It only triggers if the service state is `Critical` and attempts to restart the
service before a notification is sent.
Requirements:
* Icinga 2 as client on the remote node
* Icinga 2 service with permissions to execute Powershell scripts (which is the default)
Define an [EventCommand](9-object-types.md#objecttype-eventcommand) object `restart_service-windows`
which allows to trigger local service restarts. Put it into a [global zone](6-distributed-monitoring.md#distributed-monitoring-global-zone-config-sync)
to sync its configuration to all clients.
[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/global-templates/eventcommands.conf
object EventCommand "restart_service-windows" {
command = [
"C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe",
PluginDir + "/restart_service.ps1"
]
arguments = {
"-ServiceState" = "$service.state$"
"-ServiceStateType" = "$service.state_type$"
"-ServiceAttempt" = "$service.check_attempt$"
"-Service" = "$restart_service$"
"; exit" = {
order = 99
value = "$$LASTEXITCODE"
}
}
vars.restart_service = "$service_win_service$"
}
This event command triggers the following script which restarts the service.
The script only is executed if the service state is `CRITICAL`. Warning and Unknown states
are ignored as they indicate not an immediate failure.
Add the `restart_service.ps1` Powershell script into `C:\Program Files\Icinga2\sbin`:
param(
[string]$Service = '',
[string]$ServiceState = '',
[string]$ServiceStateType = '',
[int]$ServiceAttempt = ''
)
if (!$Service -Or !$ServiceState -Or !$ServiceStateType -Or !$ServiceAttempt) {
$scriptName = GCI $MyInvocation.PSCommandPath | Select -Expand Name;
Write-Host "USAGE: $scriptName -ServiceState servicestate -ServiceStateType servicestatetype -ServiceAttempt serviceattempt -Service service" -ForegroundColor red;
exit 3;
}
# Only restart on the third attempt of a critical event
if ($ServiceState -eq "CRITICAL" -And $ServiceStateType -eq "SOFT" -And $ServiceAttempt -eq 3) {
Restart-Service $Service;
}
exit 0;
Add a service on the master node which is executed via command endpoint on the client.
Set the `event_command` attribute to `restart_service-windows`, the name of the previously defined
EventCommand object.
[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/icinga2-client2.localdomain.conf
object Service "Service httpd" {
check_command = "service-windows"
event_command = "restart_service-windows"
max_check_attempts = 4
host_name = "icinga2-client2.localdomain"
command_endpoint = "icinga2-client2.localdomain"
vars.service_win_service = "httpd"
}
In order to test this configuration just stop the `httpd` on the remote host `icinga2-client1.localdomain`.
C:> net stop httpd
You can enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) and search for the
executed command line in `C:\ProgramData\icinga2\var\log\icinga2\debug.log`.
#### <a id="event-command-restart-service-daemon-ssh"></a> Use Event Commands to Restart Service Daemon via SSH
This example triggers a restart of the `httpd` daemon
via SSH when the `http` service check fails.
Requirements:
* SSH connection allowed (firewall, packet filters)
* icinga user with public key authentication * icinga user with public key authentication
* icinga user with sudo permissions for restarting the httpd daemon. * icinga user with sudo permissions to restart the httpd daemon.
Example on Debian: Example on Debian:
@ -1667,9 +1926,10 @@ Example on Debian:
# visudo # visudo
icinga ALL=(ALL) NOPASSWD: /etc/init.d/apache2 restart icinga ALL=(ALL) NOPASSWD: /etc/init.d/apache2 restart
Define a generic [EventCommand](9-object-types.md#objecttype-eventcommand) object `event_by_ssh` Define a generic [EventCommand](9-object-types.md#objecttype-eventcommand) object `event_by_ssh`
which can be used for all event commands triggered using ssh: which can be used for all event commands triggered using SSH:
[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/local_eventcommands.conf
/* pass event commands through ssh */ /* pass event commands through ssh */
object EventCommand "event_by_ssh" { object EventCommand "event_by_ssh" {
@ -1704,7 +1964,7 @@ is only restarted when the service is not in an `OK` state.
//only restart the daemon if state > 0 (not-ok) //only restart the daemon if state > 0 (not-ok)
//requires sudo permissions for the icinga user //requires sudo permissions for the icinga user
vars.event_by_ssh_command = "test $service.state_id$ -gt 0 && sudo /etc/init.d/$event_by_ssh_service$ restart" vars.event_by_ssh_command = "test $service.state_id$ -gt 0 && sudo systemctl restart $event_by_ssh_service$"
} }

1
doc/9-object-types.md
View File

@ -466,6 +466,7 @@ Configuration Attributes:
Command arguments can be used the same way as for [CheckCommand objects](9-object-types.md#objecttype-checkcommand-arguments). Command arguments can be used the same way as for [CheckCommand objects](9-object-types.md#objecttype-checkcommand-arguments).
More advanced examples for event command usage can be found [here](3-monitoring-basics.md#event-commands).
## <a id="objecttype-externalcommandlistener"></a> ExternalCommandListener ## <a id="objecttype-externalcommandlistener"></a> ExternalCommandListener