Dev: Allow to delay main worker process in order to attach with LLDB

Introduce `-DInternal.DebugWorkerDelay=120` and sleep inside `RunWorker()`.

Rationale: With 2.11 we've introduced a real umbrella process where the
main process is spawned as child fork. Running icinga2 in foreground on
a macOS shell with LLDB will now exit, and not follow the child process.

LLDB doesn't support `follow-fork-mode child` like GDB and therefore we
need to:

- Print the child process PID
- Sleep for X seconds to allow the developer to run `lldb -p <PID>`, set breakpoints, etc.

This commit also documents all available debug build enabled internal constants.

doc/17-language-reference.md

@@ -566,6 +566,28 @@ ICINGA2\_RLIMIT\_FILES     |**Read-write.** Defines the resource limit for `RLIM
 ICINGA2\_RLIMIT\_PROCESSES |**Read-write.** Defines the resource limit for `RLIMIT_NPROC` that should be set at start-up. Value cannot be set lower than the default `16 * 1024`. 0 disables the setting. Set in Icinga 2 sysconfig.
 ICINGA2\_RLIMIT\_STACK     |**Read-write.** Defines the resource limit for `RLIMIT_STACK` that should be set at start-up. Value cannot be set lower than the default `256 * 1024`. 0 disables the setting. Set in Icinga 2 sysconfig.
 
+#### Debug Constants and Variables <a id="icinga-constants-debug"></a>
+
+These constants are only available in debug builds for developers and help with tracing messages and attaching to debuggers.
+
+Variable                   | Description
+---------------------------|-------------------
+Internal.DebugJsonRpc      | **Read-write.** Setting this to `1` prints the raw JSON-RPC message to STDOUT.
+Internal.DebugWorkerDelay  | **Read-write.** Delays the main worker process by X seconds after forked from the umbrella process. This helps with attaching LLDB which cannot follow child forks like GDB.
+
+Example:
+
+```
+$ icinga2 daemon -DInternal.DebugWorkerDelay=120
+Closed FD 6 which we inherited from our parent process.
+[2020-01-29 12:22:33 +0100] information/cli: Icinga application loader (version: v2.11.0-477-gfe8701d77; debug)
+[2020-01-29 12:22:33 +0100] information/RunWorker: DEBUG: Current PID: 85253. Sleeping for 120 seconds to allow lldb/gdb -p <PID> attachment.
+
+$ lldb -p 85253
+(lldb) b icinga::Checkable::ProcessCheckResult
+(lldb) c
+```
+
 
 ## Apply <a id="apply"></a>
 

lib/cli/daemoncommand.cpp

@@ -191,6 +191,24 @@ pid_t l_UmbrellaPid = 0;
 static Atomic<bool> l_AllowedToWork (false);
 #endif /* _WIN32 */
 
+#ifdef I2_DEBUG
+/**
+ * Determine whether the developer wants to delay the worker process to attach a debugger to it.
+ *
+ * @return Internal.DebugWorkerDelay double
+ */
+static double GetDebugWorkerDelay()
+{
+	Namespace::Ptr internal = ScriptGlobal::Get("Internal", &Empty);
+
+	Value vdebug;
+	if (internal && internal->Get("DebugWorkerDelay", &vdebug))
+		return Convert::ToDouble(vdebug);
+
+	return 0.0;
+}
+#endif /* I2_DEBUG */
+
 /**
  * Do the actual work (config loading, ...)
  *
@@ -203,6 +221,18 @@ static Atomic<bool> l_AllowedToWork (false);
 static inline
 int RunWorker(const std::vector<std::string>& configs, bool closeConsoleLog = false, const String& stderrFile = String())
 {
+
+#ifdef I2_DEBUG
+	double delay = GetDebugWorkerDelay();
+
+	if (delay > 0.0) {
+		Log(LogInformation, "RunWorker")
+			<< "DEBUG: Current PID: " << Utility::GetPid() << ". Sleeping for " << delay << " seconds to allow lldb/gdb -p <PID> attachment.";
+
+		Utility::Sleep(delay);
+	}
+#endif /* I2_DEBUG */
+
 	Log(LogInformation, "cli", "Loading configuration file(s).");
 
 	{