Documentation: Add configuration best practice introduction.

Fixes #6318

doc/3-monitoring-basics.md

@@ -81,7 +81,61 @@ state the host/service switches to a `HARD` state and notifications are sent.
   SOFT        | The host/service has recently changed state and is being re-checked.
 
 
-## <a id="using-templates"></a> Using Templates
+
+## <a id="configuration-best-practice"></a> Configuration Best Practice
+
+The [Getting Started](#getting-started) chapter already introduced various aspects
+of the Icinga 2 configuration language. If you are ready to configure additional
+hosts, services, notifications, dependencies, etc you should think about the
+requirements first and then decide for a possible strategy.
+
+There are many ways of creating Icinga 2 configuration objects:
+
+* Manually with your preferred editor, for example vi(m), nano, notepad, etc.
+* Generated by a configuration management tool such as Puppet, Chef, Ansible, etc.
+* A configuration addon for Icinga 2
+* A custom exporter script from your CMDB or inventory tool
+* your own.
+
+In order to find the best strategy for your own configuration, ask yourself the following questions:
+
+* Do your hosts share a common group of services (for example linux hosts with disk, load, etc checks)?
+* Only a small set of users receives notifications and escalations for all hosts/services?
+
+If you can at least answer one of these questions with yes, look for the [apply rules](#using-apply) logic
+instead of defining objects on a per host and service basis.
+
+* You are required to define specific configuration for each host/service?
+* Does your configuration generation tool already know about the host-service-relationship?
+
+Then you should look for the object specific configuration setting `host_name` etc accordingly.
+
+Finding the best files and directory tree for your configuration is up to you. Make sure that
+the [icinga2.conf](#icinga2-conf) configuration file includes them, and then think about:
+
+* tree-based on locations, hostgroups, specific host attributes with sub levels of directories.
+* flat `hosts.conf`, `services.conf`, etc files for rule based configuration.
+* generated configuration with one file per host and a global configuration for groups, users, etc.
+* one big file generated from an external application (probably a bad idea for maintaining changes).
+* your own.
+
+In either way of choosing the right strategy you should additionally check the following:
+
+* Are there any specific attributes describing the host/service you could set as `vars` custom attributes?
+You can later use them for applying assign/ignore rules, or export them into external interfaces.
+* Put hosts into hostgroups, services into servicegroups and use these attributes for your apply rules.
+* Use templates to store generic attributes for your objects and apply rules making your configuration more readable.
+Details can be found in the [using templates](#using-templates) chapter.
+* Apply rules may overlap. Keep a central place (for example, `services.conf` or `notifications.conf`) storing
+the configuration instead of defining apply rules deep in your configuration tree.
+* Every plugin used as check, notification or event command requires a `Command` definition.
+Further details can be looked up in the [check commands](#check-commands) chapter.
+
+If you happen to have further questions, do not hesitate to join the [community support channels](https://support.icinga.org)
+and ask community members for their experience and best practices.
+
+
+### <a id="using-templates"></a> Using Templates
 
 Templates may be used to apply a set of identical attributes to more than one
 object:
@@ -114,7 +168,7 @@ Objects as well as templates themselves can import an arbitrary number of
 templates. Attributes inherited from a template can be overridden in the
 object if necessary.
 
-## <a id="using-apply"></a> Apply objects based on rules
+### <a id="using-apply"></a> Apply objects based on rules
 
 Instead of assigning each object (`Service`, `Notification`, `Dependency`, `ScheduledDowntime`)
 based on attribute identifiers for example `host_name` objects can be [applied](#apply).
@@ -122,7 +176,7 @@ based on attribute identifiers for example `host_name` objects can be [applied](
 Detailed scenario examples are used in their respective chapters, for example
 [apply services with custom command arguments](#using-apply-services-command-arguments).
 
-### <a id="using-apply-services"></a> Apply Services to Hosts
+#### <a id="using-apply-services"></a> Apply Services to Hosts
 
     apply Service "load" {
       import "generic-service"
@@ -137,7 +191,7 @@ In this example the `load` service will be created as object for all hosts in th
 host group. If the `no_load_check` custom attribute is set, the host will be
 ignored.
 
-### <a id="using-apply-notifications"></a> Apply Notifications to Hosts and Services
+#### <a id="using-apply-notifications"></a> Apply Notifications to Hosts and Services
 
 Notifications are applied to specific targets (`Host` or `Service`) and work in a similar
 manner:
@@ -154,7 +208,7 @@ In this example the `mail-noc` notification will be created as object for all se
 `sla` custom attribute set to `24x7`. The notification command is set to `mail-service-notification`
 and all members of the user group `noc` will get notified.
 
-### <a id="using-apply-dependencies"></a> Apply Dependencies to Hosts and Services
+#### <a id="using-apply-dependencies"></a> Apply Dependencies to Hosts and Services
 
 Detailed examples can be found in the [dependencies](#dependencies) chapter.
 
@@ -163,7 +217,7 @@ Detailed examples can be found in the [dependencies](#dependencies) chapter.
 Detailed examples can be found in the [recurring downtimes](#recurring-downtimes) chapter.
 
 
-## <a id="groups"></a> Groups
+### <a id="groups"></a> Groups
 
 Groups are used for combining hosts, services, and users into
 accessible configuration attributes and views in external (web)
@@ -219,7 +273,7 @@ the user groups are associated as attributes in `Notification` objects.
       email = "ops@example.com"
     }
 
-### <a id="group-assign"></a> Group Membership Assign
+#### <a id="group-assign"></a> Group Membership Assign
 
 If there is a certain number of hosts, services or users matching a pattern
 it's reasonable to assign the group object to these members.