Merge pull request #6670 from Icinga/feature/docs-technical-concepts-config-compiler

Add technical concepts for the config compiler and daemon CLI command

doc/19-technical-concepts.md

@@ -1,19 +1,169 @@
 # Technical Concepts <a id="technical-concepts"></a>
 
-This chapter provides insights into specific Icinga 2
+This chapter provides technical concepts and design insights
-components, libraries, features and any other technical concept
+into specific Icinga 2 components such as:
-and design.
+
+* [Application](19-technical-concepts.md#technical-concepts-application)
+* [Configuration](19-technical-concepts.md#technical-concepts-configuration)
+* [Features](19-technical-concepts.md#technical-concepts-features)
+* [Cluster](19-technical-concepts.md#technical-concepts-cluster)
+* [TLS Network IO](19-technical-concepts.md#technical-concepts-tls-network-io)
 
-<!--
 ## Application <a id="technical-concepts-application"></a>
 
-### Libraries <a id="technical-concepts-application-libraries"></a>
+### CLI Commands <a id="technical-concepts-application-cli-commands"></a>
 
+The Icinga 2 application is managed with different CLI sub commands.
+`daemon` takes care about loading the configuration files, running the
+application as daemon, etc.
+Other sub commands allow to enable features, generate and request
+TLS certificates or enter the debug console.
+
+The main entry point for each CLI command parses the command line
+parameters and then triggers the required actions.
+
+### daemon CLI command <a id="technical-concepts-application-cli-commands-daemon"></a>
+
+This CLI command loads the configuration files, starting with `icinga2.conf`.
+The [configuration compiler](19-technical-concepts.md#technical-concepts-configuration) parses the
+file and detects additional file includes, constants, and any other DSL
+specific declaration.
+
+At this stage, the configuration will already be checked against the
+defined grammar in the scanner, and custom object validators will also be
+checked.
+
+If the user provided `-C/--validate`, the CLI command returns with the
+validation exit code.
+
+When running as daemon, additional parameters are checked, e.g. whether
+this application was triggered by a reload, needs to daemonize with fork()
+involved and update the object's authority. The latter is important for
+HA-enabled cluster zones.
 
 ## Configuration <a id="technical-concepts-configuration"></a>
 
+### Lexer <a id="technical-concepts-configuration-lexer"></a>
+
+The lexer stage does not understand the DSL itself, it only
+maps specific character sequences into identifiers.
+
+This allows Icinga to detect the beginning of a string with `"`,
+reading the following characters and determining the end of the
+string with again `"`.
+
+Other parts covered by the lexer a escape sequences insides a string,
+e.g. `"\"abc"`.
+
+The lexer also identifiers logical operators, e.g. `&` or `in`,
+specific keywords like `object`, `import`, etc. and comment blocks.
+
+Please check `lib/config/config_lexer.ll` for details.
+
+Icinga uses [Flex](https://github.com/westes/flex) in the first stage.
+
+> Flex (The Fast Lexical Analyzer)
+>
+> Flex is a fast lexical analyser generator. It is a tool for generating programs
+> that perform pattern-matching on text. Flex is a free (but non-GNU) implementation
+> of the original Unix lex program.
+
+### Parser <a id="technical-concepts-configuration-parser"></a>
+
+The parser stage puts the identifiers from the lexer into more
+context with flow control and sequences.
+
+The following comparison is parsed into a left term, an operator
+and a right term.
+
+```
+x > 5
+```
+
+The DSL contains many elements which require a specific order,
+and sometimes only a left term for example.
+
+The parser also takes care of parsing an object declaration for
+example. It already knows from the lexer that `object` marks the
+beginning of an object. It then expects a type string afterwards,
+and the object name - which can be either a string with double quotes
+or a previously defined constant.
+
+An opening bracket `{` in this specific context starts the object
+scope, which also is stored for later scope specific variable access.
+
+If there's an apply rule defined, this follows the same principle.
+The config parser detects the scope of an apply rule and generates
+Icinga 2 C++ code for the parsed string tokens.
+
+```
+assign where host.vars.sla == "24x7"
+```
+
+is parsed into an assign token identifier, and the string expression
+is compiled into a new `ApplyExpression` object.
+
+The flow control inside the parser ensures that for example `ignore where`
+can only be defined when a previous `assign where` was given - or when
+inside an apply for rule.
+
+Another example are specific object types which allow assign expression,
+specifically group objects. Others objects must throw a configuration error.
+
+Please check `lib/config/config_parser.yy` for more details,
+and the [language reference](17-language-reference.md#language-reference) chapter for
+documented DSL keywords and sequences.
+
+> Icinga uses [Bison](https://en.wikipedia.org/wiki/GNU_bison) as parser generator
+> which reads a specification of a context-free language, warns about any parsing
+> ambiguities, and generates a parser in C++ which reads sequences of tokens and
+> decides whether the sequence conforms to the syntax specified by the grammar.
+
+
 ### Compiler <a id="technical-concepts-configuration-compiler"></a>
--->
+
+The config compiler initializes the scanner inside the [lexer](19-technical-concepts.md#technical-concepts-configuration-lexer)
+stage.
+
+The configuration files are parsed into memory from inside the [daemon CLI command](19-technical-concepts.md#technical-concepts-application-cli-commands-daemon)
+which invokes the config validation in `ValidateConfigFiles()`. This compiles the
+files into an AST expression which is executed.
+
+At this stage, the expressions generate so-called "config items" which
+are a pre-stage of the later compiled object.
+
+`ConfigItem::CommitItems` takes care of committing the items, and doing a
+rollback on failure. It also checks against matching apply rules from the previous run
+and generates statistics about the objects which can be seen by the config validation.
+
+`ConfigItem::CommitNewItems` collects the registered types and items,
+and checks for a specific required order, e.g. a service object needs
+a host object first.
+
+The following stages happen then:
+
+- **Commit**: A workqueue then commits the items in a parallel fashion for this specific type. The object gets its name, and the AST expression is executed. It is then registered into the item into `m_Object` as reference.
+- **OnAllConfigLoaded**: Special signal for each object to pre-load required object attributes, resolve group membership, initialize functions and timers.
+- **CreateChildObjects**: Run apply rules for this specific type.
+- **CommitNewItems**: Apply rules may generate new config items, this is to ensure that they again run through the stages.
+
+Note that the items are now committed and the configuration is validated and loaded
+into memory. The final config objects are not yet activated though.
+
+This only happens after the validation, when the application is about to be run
+with `ConfigItem::ActivateItems`.
+
+Each item has an object created in `m_Object` which is checked in a loop.
+Again, the dependency order of activated objects is important here, e.g. logger features come first, then
+config objects and last the checker, api, etc. features. This is done by sorting the objects
+based on their type specific activation priority.
+
+The following signals are triggered in the stages:
+
+- **PreActivate**: Setting the `active` flag for the config object.
+- **Activate**: Calls `Start()` on the object, sets the local HA authority and notifies subscribers that this object is now activated (e.g. for config updates in the DB backend).
+
+
 
 ## Features <a id="technical-concepts-features"></a>