Improve documentation and Systemd config for TasksMax setting

This also disables setting a value by default.

Packaging should decide to install the config file:
`/etc/systemd/system/icinga2.service.d`

fixes #5611

Signed-off-by: Michael Friedrich <michael.friedrich@icinga.com>

INSTALL.md

@@ -272,15 +272,25 @@ components (e.g. for check execution, notifications, etc.):
     [2016-12-08 16:44:25 +0100] information/ConfigItem: Committing config item(s).
     ...
 
+### Init Script
+
 Icinga 2 can be started as a daemon using the provided init script:
 
     # /etc/init.d/icinga2
     Usage: /etc/init.d/icinga2 {start|stop|restart|reload|checkconfig|status}
 
-If your distribution uses systemd:
+### Systemd
+
+If your distribution uses Systemd:
 
     # systemctl {start|stop|reload|status|enable|disable} icinga2
 
+In case the distribution is running Systemd >227, you'll also
+need to package and install the `etc/initsystem/icinga2.service.limits.conf`
+file into `/etc/systemd/system/icinga2.service.d`.
+
+### openrc
+
 Or if your distribution uses openrc (like Alpine):
 
     # rc-service icinga2

doc/02-getting-started.md

@@ -169,6 +169,7 @@ By default Icinga 2 uses the following files and directories:
   ----------------------------------------------|------------------------------------
   /etc/icinga2                        		| Contains Icinga 2 configuration files.
   /usr/lib/systemd/system/icinga2.service 	| The Icinga 2 Systemd service file on systems using Systemd.
+  /etc/systemd/system/icinga2.service.d/limits.conf | On distributions with Systemd >227, additional service limits are required.
   /etc/init.d/icinga2                 		| The Icinga 2 init script on systems using SysVinit or OpenRC
   /usr/sbin/icinga2                   		| Shell wrapper for the Icinga 2 binary.
   /usr/lib\*/icinga2				| Libraries and the Icinga 2 binary (use `find /usr -type f -name icinga2` to locate the binary path).
@@ -295,12 +296,12 @@ By default the Icinga 2 daemon is running as `icinga` user and group
 using the init script. Using Debian packages the user and group are set to
 `nagios` for historical reasons.
 
-### systemd Service <a id="systemd-service"></a>
+### Systemd Service <a id="systemd-service"></a>
 
-Some distributions (e.g. Fedora, openSUSE and RHEL/CentOS 7) use systemd. The
+Some distributions (e.g. Fedora, openSUSE and RHEL/CentOS 7) use Systemd. The
-Icinga 2 packages automatically install the necessary systemd unit files.
+Icinga 2 packages automatically install the necessary Systemd unit files.
 
-The Icinga 2 systemd service can be (re-)started, reloaded, stopped and also
+The Icinga 2 Systemd service can be (re-)started, reloaded, stopped and also
 queried for its current status.
 
     # systemctl status icinga2
@@ -343,6 +344,11 @@ Examples:
 If you're stuck with configuration errors, you can manually invoke the
 [configuration validation](11-cli-commands.md#config-validation).
 
+> **Tip**
+>
+> If you are running into fork errors with Systemd enabled distributions,
+> please check the [troubleshooting chapter](15-troubleshooting.md#check-fork-errors).
+
 ### FreeBSD <a id="running-icinga2-freebsd"></a>
 
 On FreeBSD you need to enable icinga2 in your rc.conf

doc/15-troubleshooting.md

@@ -496,16 +496,18 @@ Solution:
 * Disable the `checker` feature on clients: `icinga2 feature disable checker`.
 * Remove the inclusion of [conf.d](04-configuring-icinga-2.md#conf-d) as suggested in the [client setup docs](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
 
-
-
 ### Check Fork Errors <a id="check-fork-errors"></a>
 
-We've learned that newer kernel versions introduce a [fork limit for cgroups](https://lwn.net/Articles/663873/)
+Newer versions of Systemd on Linux limit spawned processes for
-which is enabled in SLES 12 SP2+ for example. The default value
+services.
-for `DefaultTasksMax` in Systemd is set to `512`.
 
-Icinga 2 relies on forking child processes to execute commands
+* v227 introduces the `TasksMax` setting to units which allows to specify the spawned process limit.
-and might therefore hit this limit in larger setups.
+* v228 adds `DefaultTasksMax` in the global `systemd-system.conf` with a default setting of 512 processes.
+* v231 changes the default value to 15%
+
+This can cause problems with Icinga 2 in large environments with many
+commands executed in parallel starting with Systemd v228. Some distributions
+also may have changed the defaults.
 
 The error message could look like this:
 
@@ -514,25 +516,27 @@ The error message could look like this:
 ```
 
 In order to solve the problem, increase the value for `DefaultTasksMax`
-or set it to `infinity`:
+or set it to `infinity`.
 
 ```
-[root@icinga2-master1.localdomain /]# cp /usr/lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service
+mkdir /etc/systemd/system/icinga2.service.d
-[root@icinga2-master1.localdomain /]# vim /etc/systemd/system/icinga2.service
+cat >/etc/systemd/system/icinga2.service.d/limits.conf <<EOF
-
 [Service]
-
 DefaultTasksMax=infinity
+EOF
 
-[root@icinga2-master1.localdomain /]# systemctl daemon-reload
+systemctl daemon-reload
-[root@icinga2-master1.localdomain /]# systemctl restart icinga2
+systemctl restart icinga2
 ```
 
-Please note that this setting is available since Systemd version 226.
+An example is available inside the GitHub repository in [etc/initsystem](https://github.com/Icinga/icinga2/tree/master/etc/initsystem).
 
-> **Note**
+External Resources:
->
+
-> Icinga 2 v2.7.1 adds the setting as default.
+* [Fork limit for cgroups](https://lwn.net/Articles/663873/)
+* [Systemd changelog](https://github.com/systemd/systemd/blob/master/NEWS)
+* [Icinga 2 upstream issue](https://github.com/Icinga/icinga2/issues/5611)
+* [Systemd upstream discussion](https://github.com/systemd/systemd/issues/3211)
 
 ### Late Check Results <a id="late-check-results"></a>
 

etc/initsystem/icinga2.service.cmake

@@ -10,8 +10,17 @@ ExecStart=@CMAKE_INSTALL_FULL_SBINDIR@/icinga2 daemon -d -e ${ICINGA2_ERROR_LOG}
 PIDFile=@ICINGA2_RUNDIR@/icinga2/icinga2.pid
 ExecReload=@CMAKE_INSTALL_PREFIX@/lib/icinga2/safe-reload @ICINGA2_SYSCONFIGFILE@
 TimeoutStartSec=30m
-# Introduced in Systemd 226, defaults to 512. Icinga 2 requires more tasks (checks, notifications, etc.)
+
-DefaultTasksMax=infinity
+# Systemd >228 enforces a lower process number for services.
+# Depending on the distribution and Systemd version, this must
+# be explicitly raised. Packages will set the needed values
+# into /etc/systemd/system/icinga2.service.d/limits.conf
+#
+# Please check the troubleshooting documentation for further details.
+# The values below can be used as examples for customized service files.
+
+#TasksMax=infinity
+#LimitNPROC=62883
 
 [Install]
 WantedBy=multi-user.target

etc/initsystem/icinga2.service.limits.conf

@@ -0,0 +1,9 @@
+# Icinga 2 sets Systemd default values to extend OS defaults.
+#
+# Please check the troubleshooting documentation for further details.
+
+[Service]
+TasksMax=infinity
+
+# Uncomment this setting in case of further problems.
+#LimitNPROC=62883