tyler.durden
/
icinga2
mirror of https://github.com/Icinga/icinga2.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
icinga2/doc/7-troubleshooting.md

5.8 KiB
Raw Blame History

Icinga 2 Troubleshooting

For a more verbose output of the Icinga 2 daemon increase the severity attribute of the logs to notice or debug.

Which information is required

  • Which distribution and version
  • How was Icinga 2 installed (and which repository in case)
  • Run icinga2 --version
  • Provide complete configuration snippets explaining your problem in detail
  • Provide complete logs targetting your problem
  • If the check command failed - what's the output of your manual plugin tests?
  • In case of debugging Icinga 2, the full back traces and outputs

Enable Debug Output

Run Icinga 2 in foreground with debugging enabled Specify the console log severity as additional parameter argument to -x. Default is debug.

# /usr/sbin/icinga2 -c /etc/icinga2/icinga2.conf -x notice

Additionally you can enable the debug log using

# icinga2-enable-feature debuglog
# service icinga2 restart
# tail -f /var/log/icinga2/debug.log

Checks are not executed

  • Check the debug log if the check command gets executed

  • Verify that failed depedencies do not prevent the command execution

  • Make sure that the plugin is executable by the Icinga 2 user (run a manual test)

    sudo -u icinga /usr/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 5000,100% -w 3000,80%

  • Make sure the checker feature is enabled.

    icinga2-enable-feature checker

    Total params: 1 The feature 'checker' is already enabled.

Notifications are not sent

  • Check the debug log if a notification is triggered
  • If yes, verify that all conditions are satisfied
  • Any errors on the notification command execution logged?

Verify the following configuration

  • Is the host/service the enable_notifications attribute set, and if, to which value?

  • Do the notification attributes states, types, period match the notification conditions?

  • Do the user attributes states, types, period match the notification conditions?

  • Are there any notification begin and end times configured?

  • Make sure the notification feature is enabled.

    icinga2-enable-feature notification

    Total params: 1 The feature 'notification' is already enabled.

  • Does the referenced NotificationCommand work executed as Icinga user on the shell?

Feature is not working

  • Make sure that the feature configuration is enabled by symlink from features-available/ to features-enabled and that the latter is included in icinga2.conf.
  • Are the feature attributes set correctly according to the documentation?
  • Any errors on the logs?

Configuration is ignored

  • Make sure that the line(s) are not commented (starting with // or #, or encapsulated by /* ... */.
  • Is the configuration file included in icinga2.conf?

Configuration attributes are inherited from

Icinga 2 allows you to import templates using the import keyword. If these template contain additional attributes your objects will automatically inherit them. You can override or modify these attributes in the current object.

Where are the check command definitions

Icinga 2 ships additional plugin check command definitions which are included using

include <itl>
include <plugins>

in icinga2.conf. These configurations will be overridden on upgrade, so please send modifications as proposed patches upstream. The default include path is set to LocalStateDir + "/share/icinga2/includes".

You should add your own command definitions to a new file in conf.d/ called commands.conf or similar.

Debug Icinga 2

Make sure that the debug symbols are available for Icinga 2. The Icinga 2 packages provide a debug package which must be installed separately for all involved binaries, like icinga2-bin or icinga2-ido-mysql.

# yum install icinga2-bin-debuginfo icinga2-ido-mysql-debuginfo

Compiled binaries require the -DCMAKE_BUILD_TYPE=RelWithDebInfo or -DCMAKE_BUILD_TYPE=Debug cmake build flags.

GDB

Call GDB with the binary and all arguments and run it in foreground.

# gdb --args /usr/sbin/icinga2 -c /etc/icinga2/icinga2.conf -x

GDB Run

Run the application.

(gdb) r

Kill the running application.

(gdb) k

Continue after breakpoint.

(gdb) c

GDB Backtrace

If Icinga 2 aborted its operation abnormally, generate a backtrace.

(gdb) bt
(gdb) bt full

Tip

If you're opening an issue at [https://dev.icinga.org] make sure to attach as much details as possible.

GDB Backtrace Stepping

Identifying the problem may require stepping into the backtrace analysing the current scope, attributes and possible unmet requirements. p prints the value of the selected variable or function call result.

(gdb) up
(gdb) down
(gdb) p checkable
(gdb) p checkable.px->m_Name

GDB Breakpoints

Set a breakpoint to a specific function call, or file specific line.

(gdb) b checkable.cpp:125
(gdb) b icinga::Checkable::SetEnablePerfdata

GDB will ask about loading the required symbols later, select yes instead of no.

Then run Icinga 2 until it reaches the first breakpoint. Continue with c afterwards.

(gdb) run
(gdb) c

If you want to delete all breakpoints, use d and select yes.

(gdb) d