diff --git a/INSTALL.md b/INSTALL.md index a0779cddb..6202751ba 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -272,15 +272,25 @@ components (e.g. for check execution, notifications, etc.): [2016-12-08 16:44:25 +0100] information/ConfigItem: Committing config item(s). ... +### Init Script + Icinga 2 can be started as a daemon using the provided init script: # /etc/init.d/icinga2 Usage: /etc/init.d/icinga2 {start|stop|restart|reload|checkconfig|status} -If your distribution uses systemd: +### Systemd + +If your distribution uses Systemd: # systemctl {start|stop|reload|status|enable|disable} icinga2 +In case the distribution is running Systemd >227, you'll also +need to package and install the `etc/initsystem/icinga2.service.limits.conf` +file into `/etc/systemd/system/icinga2.service.d`. + +### openrc + Or if your distribution uses openrc (like Alpine): # rc-service icinga2 diff --git a/doc/02-getting-started.md b/doc/02-getting-started.md index b45480c6d..4f4af739c 100644 --- a/doc/02-getting-started.md +++ b/doc/02-getting-started.md @@ -169,6 +169,7 @@ By default Icinga 2 uses the following files and directories: ----------------------------------------------|------------------------------------ /etc/icinga2 | Contains Icinga 2 configuration files. /usr/lib/systemd/system/icinga2.service | The Icinga 2 Systemd service file on systems using Systemd. + /etc/systemd/system/icinga2.service.d/limits.conf | On distributions with Systemd >227, additional service limits are required. /etc/init.d/icinga2 | The Icinga 2 init script on systems using SysVinit or OpenRC /usr/sbin/icinga2 | Shell wrapper for the Icinga 2 binary. /usr/lib\*/icinga2 | Libraries and the Icinga 2 binary (use `find /usr -type f -name icinga2` to locate the binary path). @@ -295,12 +296,12 @@ By default the Icinga 2 daemon is running as `icinga` user and group using the init script. Using Debian packages the user and group are set to `nagios` for historical reasons. -### systemd Service +### Systemd Service -Some distributions (e.g. Fedora, openSUSE and RHEL/CentOS 7) use systemd. The -Icinga 2 packages automatically install the necessary systemd unit files. +Some distributions (e.g. Fedora, openSUSE and RHEL/CentOS 7) use Systemd. The +Icinga 2 packages automatically install the necessary Systemd unit files. -The Icinga 2 systemd service can be (re-)started, reloaded, stopped and also +The Icinga 2 Systemd service can be (re-)started, reloaded, stopped and also queried for its current status. # systemctl status icinga2 @@ -343,6 +344,11 @@ Examples: If you're stuck with configuration errors, you can manually invoke the [configuration validation](11-cli-commands.md#config-validation). +> **Tip** +> +> If you are running into fork errors with Systemd enabled distributions, +> please check the [troubleshooting chapter](15-troubleshooting.md#check-fork-errors). + ### FreeBSD On FreeBSD you need to enable icinga2 in your rc.conf diff --git a/doc/15-troubleshooting.md b/doc/15-troubleshooting.md index cd1139c17..9058e8745 100644 --- a/doc/15-troubleshooting.md +++ b/doc/15-troubleshooting.md @@ -496,16 +496,18 @@ Solution: * Disable the `checker` feature on clients: `icinga2 feature disable checker`. * Remove the inclusion of [conf.d](04-configuring-icinga-2.md#conf-d) as suggested in the [client setup docs](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint). - - ### Check Fork Errors -We've learned that newer kernel versions introduce a [fork limit for cgroups](https://lwn.net/Articles/663873/) -which is enabled in SLES 12 SP2+ for example. The default value -for `DefaultTasksMax` in Systemd is set to `512`. +Newer versions of Systemd on Linux limit spawned processes for +services. -Icinga 2 relies on forking child processes to execute commands -and might therefore hit this limit in larger setups. +* v227 introduces the `TasksMax` setting to units which allows to specify the spawned process limit. +* v228 adds `DefaultTasksMax` in the global `systemd-system.conf` with a default setting of 512 processes. +* v231 changes the default value to 15% + +This can cause problems with Icinga 2 in large environments with many +commands executed in parallel starting with Systemd v228. Some distributions +also may have changed the defaults. The error message could look like this: @@ -514,25 +516,27 @@ The error message could look like this: ``` In order to solve the problem, increase the value for `DefaultTasksMax` -or set it to `infinity`: +or set it to `infinity`. ``` -[root@icinga2-master1.localdomain /]# cp /usr/lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service -[root@icinga2-master1.localdomain /]# vim /etc/systemd/system/icinga2.service - +mkdir /etc/systemd/system/icinga2.service.d +cat >/etc/systemd/system/icinga2.service.d/limits.conf < **Note** -> -> Icinga 2 v2.7.1 adds the setting as default. +External Resources: + +* [Fork limit for cgroups](https://lwn.net/Articles/663873/) +* [Systemd changelog](https://github.com/systemd/systemd/blob/master/NEWS) +* [Icinga 2 upstream issue](https://github.com/Icinga/icinga2/issues/5611) +* [Systemd upstream discussion](https://github.com/systemd/systemd/issues/3211) ### Late Check Results diff --git a/etc/initsystem/icinga2.service.cmake b/etc/initsystem/icinga2.service.cmake index 7279adecf..895db7149 100644 --- a/etc/initsystem/icinga2.service.cmake +++ b/etc/initsystem/icinga2.service.cmake @@ -10,8 +10,17 @@ ExecStart=@CMAKE_INSTALL_FULL_SBINDIR@/icinga2 daemon -d -e ${ICINGA2_ERROR_LOG} PIDFile=@ICINGA2_RUNDIR@/icinga2/icinga2.pid ExecReload=@CMAKE_INSTALL_PREFIX@/lib/icinga2/safe-reload @ICINGA2_SYSCONFIGFILE@ TimeoutStartSec=30m -# Introduced in Systemd 226, defaults to 512. Icinga 2 requires more tasks (checks, notifications, etc.) -DefaultTasksMax=infinity + +# Systemd >228 enforces a lower process number for services. +# Depending on the distribution and Systemd version, this must +# be explicitly raised. Packages will set the needed values +# into /etc/systemd/system/icinga2.service.d/limits.conf +# +# Please check the troubleshooting documentation for further details. +# The values below can be used as examples for customized service files. + +#TasksMax=infinity +#LimitNPROC=62883 [Install] WantedBy=multi-user.target diff --git a/etc/initsystem/icinga2.service.limits.conf b/etc/initsystem/icinga2.service.limits.conf new file mode 100644 index 000000000..dea0cde02 --- /dev/null +++ b/etc/initsystem/icinga2.service.limits.conf @@ -0,0 +1,9 @@ +# Icinga 2 sets Systemd default values to extend OS defaults. +# +# Please check the troubleshooting documentation for further details. + +[Service] +TasksMax=infinity + +# Uncomment this setting in case of further problems. +#LimitNPROC=62883