icinga2/doc/7-troubleshooting.md

Icinga 2 Troubleshooting

For a more verbose output of the Icinga 2 daemon increase the severity attribute of the logs to notice or debug.

Which information is required

• Which distribution and version
• How was Icinga 2 installed (and which repository in case)
• Run icinga2 --version
• Provide complete configuration snippets explaining your problem in detail
• Provide complete logs targetting your problem
• If the check command failed - what's the output of your manual plugin tests?
• In case of debugging Icinga 2, the full back traces and outputs

Enable Debug Output

Run Icinga 2 in the foreground with debugging enabled. Specify the console log severity as an additional parameter argument to -x.

# /usr/sbin/icinga2 -c /etc/icinga2/icinga2.conf -x notice

Additionally you can enable the debug log using

# icinga2 feature enable debuglog
# service icinga2 restart
# tail -f /var/log/icinga2/debug.log

List Configuration Objects

The icinga2 object list cli command can be used to list all configuration objects and their attributes. The tool also shows where each of the attributes was modified:

# icinga2 object list

Object 'localhost!ssh' of type 'Service':
  * __name = 'localhost!ssh'
  * check_command = 'ssh'
    % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
  * check_interval = 60
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
  * host_name = 'localhost'
    % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
  * max_check_attempts = 3
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
  * name = 'ssh'
  * retry_interval = 30
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
  * templates = [ 'ssh', 'generic-service' ]
    % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
    % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
  * type = 'Service'
  * vars
    % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
    * sla = '24x7'
      % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19

[...]

You can filter by name or type too.

# icinga2 object list --name *ssh* --type Service
Object 'localhost!ssh' of type 'Service':
  * __name = 'localhost!ssh'
  * check_command = 'ssh'
    % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
  * check_interval = 60
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
  * host_name = 'localhost'
    % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
  * max_check_attempts = 3
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
  * name = 'ssh'
  * retry_interval = 30
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
  * templates = [ 'ssh', 'generic-service' ]
    % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
    % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
  * type = 'Service'
  * vars
    % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
    * sla = '24x7'
      % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19

Found 1 Service objects.

[2014-10-15 14:27:19 +0200] information/cli: Parsed 175 objects.

Checks are not executed

• Check the debug log to see if the check command gets executed
• Verify that failed depedencies do not prevent command execution
• Make sure that the plugin is executable by the Icinga 2 user (run a manual test)
• Make sure the checker feature is enabled.

Examples:

# sudo -u icinga /usr/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 5000,100% -w 3000,80%

# icinga2 feature enable checker
The feature 'checker' is already enabled.

Notifications are not sent

• Check the debug log to see if a notification is triggered
• If yes, verify that all conditions are satisfied
• Are any errors on the notification command execution logged?

Verify the following configuration

• Is the host/service the enable_notifications attribute set, and if, to which value?
• Do the notification attributes states, types, period match the notification conditions?
• Do the user attributes states, types, period match the notification conditions?
• Are there any notification begin and end times configured?
• Make sure the notification feature is enabled.
• Does the referenced NotificationCommand work when executed as Icinga user on the shell?

Examples:

# icinga2 feature enable notification
The feature 'notification' is already enabled.

Feature is not working

• Make sure that the feature configuration is enabled by symlinking from features-available/ to features-enabled and that the latter is included in icinga2.conf.
• Are the feature attributes set correctly according to the documentation?
• Any errors on the logs?

Configuration is ignored

• Make sure that the line(s) are not commented (starting with // or #, or encapsulated by /* ... */).
• Is the configuration file included in icinga2.conf?

Configuration attributes are inherited from

Icinga 2 allows you to import templates using the import keyword. If these templates contain additional attributes, your objects will automatically inherit them. You can override or modify these attributes in the current object.

Where are the check command definitions

Icinga 2 ships additional plugin check command definitions which are included using

include <itl>
include <plugins>

in icinga2.conf. These configurations will be overridden on upgrade, so please send modifications as proposed patches upstream. The default include path is set to LocalStateDir + "/share/icinga2/includes".

You should add your own command definitions to a new file in conf.d/ called commands.conf or similar.

Cluster Troubleshooting

You should configure the cluster health checks if you haven't done so already.

Note

Some problems just exist due to wrong file permissions or packet filters applied. Make sure to check these in the first place.

Cluster Troubleshooting Connection Errors

General connection errors normally lead you to one of the following problems:

• Wrong network configuration
• Packet loss on the connection
• Firewall rules preventing traffic

Use tools like netstat, tcpdump, nmap, etc to make sure that the cluster communication happens (default port is 5665).

# tcpdump -n port 5665 -i any

# netstat -tulpen | grep icinga

# nmap yourclusternode.localdomain

Cluster Troubleshooting SSL Errors

If the cluster communication fails with cryptic SSL error messages, make sure to check the following

• File permissions on the SSL certificate files
• Does the used CA match for all cluster endpoints?

Examples:

# ls -la /etc/icinga2/pki

Cluster Troubleshooting Message Errors

At some point, when the network connection is broken or gone, the Icinga 2 instances will be disconnected. If the connection can't be re-established between zones and endpoints, they remain in a Split-Brain-mode and history may differ.

Although the Icinga 2 cluster protocol stores historical events in a replay log for later synchronisation, you should make sure to check why the network connection failed.

Cluster Troubleshooting Config Sync

If the cluster zones do not sync their configuration, make sure to check the following:

• Within a config master zone, only one configuration master is allowed to have its config in /etc/icinga2/zones.d. ** The master syncs the configuration to /var/lib/icinga2/api/zones/ during startup and only syncs valid configuration to the other nodes ** The other nodes receive the configuration into /var/lib/icinga2/api/zones/
• The icinga2.log log file will indicate whether this ApiListener accepts config, or not

Debug Icinga 2

Make sure that the debug symbols are available for Icinga 2. The Icinga 2 packages provide a debug package which must be installed separately for all involved binaries, like icinga2-bin or icinga2-ido-mysql.

# yum install icinga2-debuginfo

# zypper install icinga2-bin-debuginfo icinga2-ido-mysql-debuginfo

# apt-get install icinga2-dbg

Compiled binaries require the -DCMAKE_BUILD_TYPE=RelWithDebInfo or -DCMAKE_BUILD_TYPE=Debug cmake build flags.

GDB

Call GDB with the binary and all arguments and run it in foreground.

# gdb --args /usr/sbin/icinga2 -c /etc/icinga2/icinga2.conf -x

GDB Run

Run the application.

(gdb) r

Kill the running application.

(gdb) k

Continue after breakpoint.

(gdb) c

GDB Backtrace

If Icinga 2 aborted its operation abnormally, generate a backtrace.

(gdb) bt
(gdb) bt full

Tip

If you're opening an issue at [https://dev.icinga.org] make sure to attach as much detail as possible.

GDB Backtrace Stepping

Identifying the problem may require stepping into the backtrace, analysing the current scope, attributes, and possible unmet requirements. p prints the value of the selected variable or function call result.

(gdb) up
(gdb) down
(gdb) p checkable
(gdb) p checkable.px->m_Name

GDB Breakpoints

To set a breakpoint to a specific function call, or file specific line.

(gdb) b checkable.cpp:125
(gdb) b icinga::Checkable::SetEnablePerfdata

GDB will ask about loading the required symbols later, select yes instead of no.

Then run Icinga 2 until it reaches the first breakpoint. Continue with c afterwards.

(gdb) run
(gdb) c

If you want to delete all breakpoints, use d and select yes.

(gdb) d