Merge pull request #10418 from Icinga/jschmidt/doc-impr-unity-builds-core-dumps

Improvements to core dumps section in developer documentation (and some smaller stuff)

doc/19-technical-concepts.md

@@ -204,7 +204,7 @@ You can read the full story [here](https://github.com/Icinga/icinga2/issues/7309
 
 With 2.11 you'll now see 3 processes:
 
-- The umbrella process which takes care about signal handling and process spawning/stopping
+- The umbrella process which takes care of signal handling and process spawning/stopping
 - The main process with the check scheduler, notifications, etc.
 - The execution helper process
 

doc/21-development.md

@@ -267,73 +267,130 @@ $3 = std::vector of length 11, capacity 16 = {{static NPos = 1844674407370955161
 
 ### Core Dump <a id="development-debug-core-dump"></a>
 
-When the Icinga 2 daemon crashes with a `SIGSEGV` signal
+When the Icinga 2 daemon is terminated by `SIGSEGV` or `SIGABRT`, a core dump file
-a core dump file should be written. This will help
+should be written. This will help developers to analyze and fix the problem.
-developers to analyze and fix the problem.
 
-#### Core Dump File Size Limit <a id="development-debug-core-dump-limit"></a>
+#### Core Dump Kernel Pattern <a id="development-debug-core-dump-format"></a>
 
-This requires setting the core dump file size to `unlimited`.
+Core dumps are generated according to the format specified in
+`/proc/sys/kernel/core_pattern`. This can either be a path relative to the
+directory the program was started in, an absolute path or a pipe to a different
+program.
 
+For more information see the [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) man page.
 
-##### Systemd
+#### Systemd Coredumpctl <a id="development-debug-core-dump-systemd"></a>
 
+Most distributions offer systemd's coredumpctl either by default or as a package.
+Distributions that offer it by default include RHEL and SLES, on others like
+Debian or Ubuntu it can be installed via the `systemd-coredump` package.
+When set up correctly, `core_pattern` will look something like this:
 ```
-systemctl edit icinga2.service
+# cat /proc/sys/kernel/core_pattern
-
+|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h`
-[Service]
-...
-LimitCORE=infinity
-
-systemctl daemon-reload
-
-systemctl restart icinga2
 ```
 
-##### Init Script
+You can look at the generated core dumps with the `coredumpctl list` command.
+You can show information, including a stack trace using
+`coredumpctl show icinga2 -1` and retrieve the actual core dump file with
+`coredumpctl dump icinga2 -1 --output <file>`.
 
+For further information on how to configure and use coredumpctl, read the man pages
+[coredumpctl(1)](https://man7.org/linux/man-pages/man1/coredumpctl.1.html) and
+[coredump.conf(5)](https://man7.org/linux/man-pages/man5/coredump.conf.5.html).
+
+#### Ubuntu Apport <a id="development-debug-core-dump-apport"></a>
+
+Ubuntu uses their own application `apport` to record core dumps. When it is
+enabled, your `core_pattern` will look like this:
 ```
-vim /etc/init.d/icinga2
+# cat /proc/sys/kernel/core_pattern
-...
+|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
-ulimit -c unlimited
-
-service icinga2 restart
 ```
 
-##### Verify
+Apport is unsuitable for development work, because by default it only works
-
+with Ubuntu packages and it has a rather complicated interface for retrieving
-Verify that the Icinga 2 process core file size limit is set to `unlimited`.
+the core dump. So unless you rely on Apport for some other workflow, systemd's
+coredumpctl is a much better option and is available on Ubuntu in the
+`systemd-coredump` package that can replace Apport on your system with no
+further setup required.
 
+If you still want to use Apport however, to set it up to work with unpackaged programs,
+add the following (create the file if it doesn't exist) to `/etc/apport/settings`:
 ```
-for pid in $(pidof icinga2); do cat /proc/$pid/limits; done
+[main]
-
+unpackaged=true
-...
+```
-Max core file size        unlimited            unlimited            bytes
+and restart Apport:
+```
+systemctl restart apport.service
 ```
 
+When the program crashes you can then find an Apport crash report in `/var/crash/`
+that you can read with the interactive `apport-cli` command. To extract the core
+dump you run `apport-unpack /var/crash/<crash-file> <output-dir>` which then
+saves a `<outputdir>/CoreDump` file that contains the actual core dump.
 
-#### Core Dump Kernel Format <a id="development-debug-core-dump-format"></a>
+#### Directly to a File <a id="development-debug-core-dump-direct"></a>
 
-The Icinga 2 daemon runs with the SUID bit set. Therefore you need
+If coredumpctl is not available, simply writing the core dump directly to a file
-to explicitly enable core dumps for SUID on Linux.
+is also sufficient. You can set up your `core_pattern` to write a file to a
+suitable path:
 
 ```bash
-sysctl -w fs.suid_dumpable=2
+sysctl -w kernel.core_pattern=/var/lib/cores/core.%e.%p.%h.%t
-```
-
-Adjust the coredump kernel format and file location on Linux:
-
-```bash
-sysctl -w kernel.core_pattern=/var/lib/cores/core.%e.%p
-
 install -m 1777 -d /var/lib/cores
 ```
 
-MacOS:
+If you want to make this setting permanent you can also add a file to
+`/etc/sysctl.d`, named something like `80-coredumps.conf`:
+```
+kernel.core_pattern = /var/lib/cores/core.%e.%p.%h.%t
+```
+
+This will create core dump files in `/var/lib/cores` where `%e` is the truncated
+name of the program, `%p` is the programs PID, `%h` is the hostname, and `%t` a
+timestamp.
+
+Note that unlike the other methods this requires the core size limit to be set
+for the process. When starting Icinga 2 via systemd you can set it to unlimited
+by adding the following to `/etc/systemd/system/icinga2.service.d/limits.conf`:
+```
+[Service]
+LimitCORE=infinity
+```
+
+Then reload and restart icinga:
+```bash
+systemctl daemon-reload
+systemctl restart icinga2.service
+```
+
+Alternatively you edit and reload in one step:
+```bash
+systemctl edit --drop-in=limits icinga2.service`
+```
+
+When using an init script or starting manually, you need to run `ulimit -c unlimited`
+before starting the program:
+```bash
+ulimit -c unlimited
+./icinga2 daemon
+```
+
+To verify that the limit has been set to `unlimited` run the following:
+```bash
+for pid in $(pidof icinga2); do cat /proc/$pid/limits; done
+```
+And look for the line:
+```
+Max core file size        unlimited            unlimited            bytes
+```
+
+#### MacOS <a id="development-debug-core-dump-macos"></a>
 
 ```bash
 sysctl -w kern.corefile=/cores/core.%P
-
 chmod 777 /cores
 ```
 
@@ -683,7 +740,7 @@ these tools:
 - vim
 - CLion (macOS, Linux)
 - MS Visual Studio (Windows)
-- Atom
+- Emacs
 
 Editors differ on the functionality. The more helpers you get for C++ development,
 the faster your development workflow will be.
@@ -741,12 +798,12 @@ perfdata       | Performance data related, including Graphite, Elastic, etc.
 db\_ido        | IDO database abstraction layer.
 db\_ido\_mysql | IDO database driver for MySQL.
 db\_ido\_pgsql | IDO database driver for PgSQL.
-mysql\_shin    | Library stub for linking against the MySQL client libraries.
+mysql\_shim    | Library stub for linking against the MySQL client libraries.
 pgsql\_shim    | Library stub for linking against the PgSQL client libraries.
 
 #### Class Compiler <a id="development-develop-design-patterns-class-compiler"></a>
 
-Another thing you will recognize are the `.ti` files which are compiled
+Something else you might notice are the `.ti` files which are compiled
 by our own class compiler into actual source code. The meta language allows
 developers to easily add object attributes and specify their behaviour.
 
@@ -792,17 +849,18 @@ The most common benefits:
 
 #### Unity Builds <a id="development-develop-builds-unity-builds"></a>
 
-Another thing you should be aware of: Unity builds on and off.
+You should be aware that by default unity builds are enabled. You can turn them
+off by setting the `ICINGA2_UNITY_BUILD` CMake option to `OFF`.
 
 Typically, we already use caching mechanisms to reduce recompile time with ccache.
 For release builds, there's always a new build needed as the difference is huge compared
 to a previous (major) release.
 
-Therefore we've invented the Unity builds, which basically concatenates all source files
+Unity builds basically concatenate all source files into one big library source code file.
-into one big library source code file. The compiler then doesn't need to load the many small
+The compiler then doesn't need to load many small files, each with all of their includes,
-files but compiles and links this huge one.
+but compiles and links only a few huge ones.
 
-Unity builds require more memory which is why you should disable them for development
+However, unity builds require more memory which is why you should disable them for development
 builds in small sized VMs (Linux, Windows) and also Docker containers.
 
 There's a couple of header files which are included everywhere. If you touch/edit them,
@@ -1228,7 +1286,7 @@ every second.
 
 Avoid log messages which could irritate the user. During
 implementation, developers can change log levels to better
-see what's going one, but remember to change this back to `debug`
+see what's going on, but remember to change this back to `debug`
 or remove it entirely.
 
 
@@ -2262,7 +2320,7 @@ cmake .. -DCMAKE_INSTALL_PREFIX=/tmp/icinga2
 
 ### CMake Variables <a id="development-package-builds-cmake-variables"></a>
 
-In addition to `CMAKE_INSTALL_PREFIX` here are most of the supported Icinga-specific cmake variables.
+In addition to `CMAKE_INSTALL_PREFIX` here are most of the supported Icinga-specific CMake variables.
 
 For all variables regarding defaults paths on in CMake, see
 [GNUInstallDirs](https://cmake.org/cmake/help/latest/module/GNUInstallDirs.html).