mirror of
https://github.com/Icinga/icinga2.git
synced 2025-07-22 05:04:30 +02:00
Merge pull request #10418 from Icinga/jschmidt/doc-impr-unity-builds-core-dumps
Improvements to core dumps section in developer documentation (and some smaller stuff)
This commit is contained in:
Julian Brost
GitHub
commit
33824c2acc
@ -204,7 +204,7 @@ You can read the full story [here](https://github.com/Icinga/icinga2/issues/7309
|
||||
|
||||
With 2.11 you'll now see 3 processes:
|
||||
|
||||
- The umbrella process which takes care about signal handling and process spawning/stopping
|
||||
- The umbrella process which takes care of signal handling and process spawning/stopping
|
||||
- The main process with the check scheduler, notifications, etc.
|
||||
- The execution helper process
|
||||
|
||||
|
||||
@ -267,73 +267,130 @@ $3 = std::vector of length 11, capacity 16 = {{static NPos = 1844674407370955161
|
||||
|
||||
### Core Dump <a id="development-debug-core-dump"></a>
|
||||
|
||||
When the Icinga 2 daemon crashes with a `SIGSEGV` signal
|
||||
a core dump file should be written. This will help
|
||||
developers to analyze and fix the problem.
|
||||
When the Icinga 2 daemon is terminated by `SIGSEGV` or `SIGABRT`, a core dump file
|
||||
should be written. This will help developers to analyze and fix the problem.
|
||||
|
||||
#### Core Dump File Size Limit <a id="development-debug-core-dump-limit"></a>
|
||||
#### Core Dump Kernel Pattern <a id="development-debug-core-dump-format"></a>
|
||||
|
||||
This requires setting the core dump file size to `unlimited`.
|
||||
Core dumps are generated according to the format specified in
|
||||
`/proc/sys/kernel/core_pattern`. This can either be a path relative to the
|
||||
directory the program was started in, an absolute path or a pipe to a different
|
||||
program.
|
||||
|
||||
For more information see the [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) man page.
|
||||
|
||||
##### Systemd
|
||||
#### Systemd Coredumpctl <a id="development-debug-core-dump-systemd"></a>
|
||||
|
||||
Most distributions offer systemd's coredumpctl either by default or as a package.
|
||||
Distributions that offer it by default include RHEL and SLES, on others like
|
||||
Debian or Ubuntu it can be installed via the `systemd-coredump` package.
|
||||
When set up correctly, `core_pattern` will look something like this:
|
||||
```
|
||||
systemctl edit icinga2.service
|
||||
|
||||
[Service]
|
||||
...
|
||||
LimitCORE=infinity
|
||||
|
||||
systemctl daemon-reload
|
||||
|
||||
systemctl restart icinga2
|
||||
# cat /proc/sys/kernel/core_pattern
|
||||
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h`
|
||||
```
|
||||
|
||||
##### Init Script
|
||||
You can look at the generated core dumps with the `coredumpctl list` command.
|
||||
You can show information, including a stack trace using
|
||||
`coredumpctl show icinga2 -1` and retrieve the actual core dump file with
|
||||
`coredumpctl dump icinga2 -1 --output <file>`.
|
||||
|
||||
For further information on how to configure and use coredumpctl, read the man pages
|
||||
[coredumpctl(1)](https://man7.org/linux/man-pages/man1/coredumpctl.1.html) and
|
||||
[coredump.conf(5)](https://man7.org/linux/man-pages/man5/coredump.conf.5.html).
|
||||
|
||||
#### Ubuntu Apport <a id="development-debug-core-dump-apport"></a>
|
||||
|
||||
Ubuntu uses their own application `apport` to record core dumps. When it is
|
||||
enabled, your `core_pattern` will look like this:
|
||||
```
|
||||
vim /etc/init.d/icinga2
|
||||
...
|
||||
ulimit -c unlimited
|
||||
|
||||
service icinga2 restart
|
||||
# cat /proc/sys/kernel/core_pattern
|
||||
|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
|
||||
```
|
||||
|
||||
##### Verify
|
||||
|
||||
Verify that the Icinga 2 process core file size limit is set to `unlimited`.
|
||||
Apport is unsuitable for development work, because by default it only works
|
||||
with Ubuntu packages and it has a rather complicated interface for retrieving
|
||||
the core dump. So unless you rely on Apport for some other workflow, systemd's
|
||||
coredumpctl is a much better option and is available on Ubuntu in the
|
||||
`systemd-coredump` package that can replace Apport on your system with no
|
||||
further setup required.
|
||||
|
||||
If you still want to use Apport however, to set it up to work with unpackaged programs,
|
||||
add the following (create the file if it doesn't exist) to `/etc/apport/settings`:
|
||||
```
|
||||
for pid in $(pidof icinga2); do cat /proc/$pid/limits; done
|
||||
|
||||
...
|
||||
Max core file size unlimited unlimited bytes
|
||||
[main]
|
||||
unpackaged=true
|
||||
```
|
||||
and restart Apport:
|
||||
```
|
||||
systemctl restart apport.service
|
||||
```
|
||||
|
||||
When the program crashes you can then find an Apport crash report in `/var/crash/`
|
||||
that you can read with the interactive `apport-cli` command. To extract the core
|
||||
dump you run `apport-unpack /var/crash/<crash-file> <output-dir>` which then
|
||||
saves a `<outputdir>/CoreDump` file that contains the actual core dump.
|
||||
|
||||
#### Core Dump Kernel Format <a id="development-debug-core-dump-format"></a>
|
||||
#### Directly to a File <a id="development-debug-core-dump-direct"></a>
|
||||
|
||||
The Icinga 2 daemon runs with the SUID bit set. Therefore you need
|
||||
to explicitly enable core dumps for SUID on Linux.
|
||||
If coredumpctl is not available, simply writing the core dump directly to a file
|
||||
is also sufficient. You can set up your `core_pattern` to write a file to a
|
||||
suitable path:
|
||||
|
||||
```bash
|
||||
sysctl -w fs.suid_dumpable=2
|
||||
```
|
||||
|
||||
Adjust the coredump kernel format and file location on Linux:
|
||||
|
||||
```bash
|
||||
sysctl -w kernel.core_pattern=/var/lib/cores/core.%e.%p
|
||||
|
||||
sysctl -w kernel.core_pattern=/var/lib/cores/core.%e.%p.%h.%t
|
||||
install -m 1777 -d /var/lib/cores
|
||||
```
|
||||
|
||||
MacOS:
|
||||
If you want to make this setting permanent you can also add a file to
|
||||
`/etc/sysctl.d`, named something like `80-coredumps.conf`:
|
||||
```
|
||||
kernel.core_pattern = /var/lib/cores/core.%e.%p.%h.%t
|
||||
```
|
||||
|
||||
This will create core dump files in `/var/lib/cores` where `%e` is the truncated
|
||||
name of the program, `%p` is the programs PID, `%h` is the hostname, and `%t` a
|
||||
timestamp.
|
||||
|
||||
Note that unlike the other methods this requires the core size limit to be set
|
||||
for the process. When starting Icinga 2 via systemd you can set it to unlimited
|
||||
by adding the following to `/etc/systemd/system/icinga2.service.d/limits.conf`:
|
||||
```
|
||||
[Service]
|
||||
LimitCORE=infinity
|
||||
```
|
||||
|
||||
Then reload and restart icinga:
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
systemctl restart icinga2.service
|
||||
```
|
||||
|
||||
Alternatively you edit and reload in one step:
|
||||
```bash
|
||||
systemctl edit --drop-in=limits icinga2.service`
|
||||
```
|
||||
|
||||
When using an init script or starting manually, you need to run `ulimit -c unlimited`
|
||||
before starting the program:
|
||||
```bash
|
||||
ulimit -c unlimited
|
||||
./icinga2 daemon
|
||||
```
|
||||
|
||||
To verify that the limit has been set to `unlimited` run the following:
|
||||
```bash
|
||||
for pid in $(pidof icinga2); do cat /proc/$pid/limits; done
|
||||
```
|
||||
And look for the line:
|
||||
```
|
||||
Max core file size unlimited unlimited bytes
|
||||
```
|
||||
|
||||
#### MacOS <a id="development-debug-core-dump-macos"></a>
|
||||
|
||||
```bash
|
||||
sysctl -w kern.corefile=/cores/core.%P
|
||||
|
||||
chmod 777 /cores
|
||||
```
|
||||
|
||||
@ -683,7 +740,7 @@ these tools:
|
||||
- vim
|
||||
- CLion (macOS, Linux)
|
||||
- MS Visual Studio (Windows)
|
||||
- Atom
|
||||
- Emacs
|
||||
|
||||
Editors differ on the functionality. The more helpers you get for C++ development,
|
||||
the faster your development workflow will be.
|
||||
@ -741,12 +798,12 @@ perfdata | Performance data related, including Graphite, Elastic, etc.
|
||||
db\_ido | IDO database abstraction layer.
|
||||
db\_ido\_mysql | IDO database driver for MySQL.
|
||||
db\_ido\_pgsql | IDO database driver for PgSQL.
|
||||
mysql\_shin | Library stub for linking against the MySQL client libraries.
|
||||
mysql\_shim | Library stub for linking against the MySQL client libraries.
|
||||
pgsql\_shim | Library stub for linking against the PgSQL client libraries.
|
||||
|
||||
#### Class Compiler <a id="development-develop-design-patterns-class-compiler"></a>
|
||||
|
||||
Another thing you will recognize are the `.ti` files which are compiled
|
||||
Something else you might notice are the `.ti` files which are compiled
|
||||
by our own class compiler into actual source code. The meta language allows
|
||||
developers to easily add object attributes and specify their behaviour.
|
||||
|
||||
@ -792,17 +849,18 @@ The most common benefits:
|
||||
|
||||
#### Unity Builds <a id="development-develop-builds-unity-builds"></a>
|
||||
|
||||
Another thing you should be aware of: Unity builds on and off.
|
||||
You should be aware that by default unity builds are enabled. You can turn them
|
||||
off by setting the `ICINGA2_UNITY_BUILD` CMake option to `OFF`.
|
||||
|
||||
Typically, we already use caching mechanisms to reduce recompile time with ccache.
|
||||
For release builds, there's always a new build needed as the difference is huge compared
|
||||
to a previous (major) release.
|
||||
|
||||
Therefore we've invented the Unity builds, which basically concatenates all source files
|
||||
into one big library source code file. The compiler then doesn't need to load the many small
|
||||
files but compiles and links this huge one.
|
||||
Unity builds basically concatenate all source files into one big library source code file.
|
||||
The compiler then doesn't need to load many small files, each with all of their includes,
|
||||
but compiles and links only a few huge ones.
|
||||
|
||||
Unity builds require more memory which is why you should disable them for development
|
||||
However, unity builds require more memory which is why you should disable them for development
|
||||
builds in small sized VMs (Linux, Windows) and also Docker containers.
|
||||
|
||||
There's a couple of header files which are included everywhere. If you touch/edit them,
|
||||
@ -1228,7 +1286,7 @@ every second.
|
||||
|
||||
Avoid log messages which could irritate the user. During
|
||||
implementation, developers can change log levels to better
|
||||
see what's going one, but remember to change this back to `debug`
|
||||
see what's going on, but remember to change this back to `debug`
|
||||
or remove it entirely.
|
||||
|
||||
|
||||
@ -2262,7 +2320,7 @@ cmake .. -DCMAKE_INSTALL_PREFIX=/tmp/icinga2
|
||||
|
||||
### CMake Variables <a id="development-package-builds-cmake-variables"></a>
|
||||
|
||||
In addition to `CMAKE_INSTALL_PREFIX` here are most of the supported Icinga-specific cmake variables.
|
||||
In addition to `CMAKE_INSTALL_PREFIX` here are most of the supported Icinga-specific CMake variables.
|
||||
|
||||
For all variables regarding defaults paths on in CMake, see
|
||||
[GNUInstallDirs](https://cmake.org/cmake/help/latest/module/GNUInstallDirs.html).
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user