diff --git a/doc/16-troubleshooting.md b/doc/16-troubleshooting.md index 99ad87789..08f27970d 100644 --- a/doc/16-troubleshooting.md +++ b/doc/16-troubleshooting.md @@ -37,6 +37,10 @@ and `debug`. The `icinga2 object list` CLI command can be used to list all configuration objects and their attributes. The tool also shows where each of the attributes was modified. +> **Tip** +> +> Use the Icinga 2 API to access [config objects at runtime](9-icinga2-api.md#icinga2-api-config-objects) directly. + That way you can also identify which objects have been created from your [apply rules](18-language-reference.md#apply). # icinga2 object list @@ -115,6 +119,7 @@ or similar. * Verify that failed depedencies do not prevent command execution * Make sure that the plugin is executable by the Icinga 2 user (run a manual test) * Make sure the [checker](8-cli-commands.md#enable-features) feature is enabled. +* Use the Icinga 2 API [event streams](9-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams. Examples: @@ -123,6 +128,10 @@ Examples: # icinga2 feature enable checker The feature 'checker' is already enabled. +Fetch all check result events matching the `event.service` name `random`: + + $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugchecks&types=CheckResult&filter=match%28%22random*%22,event.service%29' + ## Notifications are not sent @@ -139,15 +148,22 @@ Verify the following configuration * Make sure the [notification](8-cli-commands.md#enable-features) feature is enabled. * Does the referenced NotificationCommand work when executed as Icinga user on the shell? -If notifications are to be sent via mail make sure that the mail program specified exists. +If notifications are to be sent via mail make sure that the mail program specified inside the +[NotificationCommand object](6-object-types.md#objecttype-notificationcommand) exists. The name and location depends on the distribution so the preconfigured setting might have to be changed on your system. + Examples: # icinga2 feature enable notification The feature 'notification' is already enabled. +You can use the Icinga 2 API [event streams](9-icinga2-api.md#icinga2-api-event-streams) to receive live notification streams: + + $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugnotifications&types=Notification' + + ## Feature is not working * Make sure that the feature configuration is enabled by symlinking from `features-available/` @@ -175,7 +191,7 @@ did not properly escape the single dollar sign preventing its usage as [runtime critical/config: Error: Validation failed for Object 'ping4' (Type: 'Service') at /etc/icinga2/zones.d/global-templates/windows.conf:24: Closing $ not found in macro format string 'top-syntax=${list}'. -## Cluster Troubleshooting +## Cluster and Clients Troubleshooting This applies to anything using the cluster protocol: @@ -286,7 +302,7 @@ On SLES11 you'll need to use the `openssl1` command instead of `openssl`. ### Cluster Troubleshooting Message Errors At some point, when the network connection is broken or gone, the Icinga 2 instances -will be disconnected. If the connection can't be re-established between zones and endpoints, +will be disconnected. If the connection can't be re-established between endpoints in the same HA zone, they remain in a Split-Brain-mode and history may differ. Although the Icinga 2 cluster protocol stores historical events in a [replay log](16-troubleshooting.md#troubleshooting-cluster-replay-log) @@ -306,8 +322,17 @@ the following (e.g. by invoking a forced check from the web interface): * Referenced check plugin not found on the remote client. * Runtime warnings and errors, e.g. unresolved runtime macros or configuration problems. * Specific error messages are also populated into `UNKNOWN` check results including a detailed error message in their output. +* Verify the `check_source` object attribute. This is populated by the node executing the check. * More verbose logs are found inside the [debug log](16-troubleshooting.md#troubleshooting-enable-debug-output). +* Use the Icinga 2 API [event streams](9-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams. + +Fetch all check result events matching the `event.service` name `remote-client`: + + $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugcommandendpoint&types=CheckResult&filter=match%28%22remote-client*%22,event.service%29' + + + ### Cluster Troubleshooting Config Sync If the cluster zones do not sync their configuration, make sure to check the following: @@ -318,6 +343,9 @@ If the cluster zones do not sync their configuration, make sure to check the fol * The `icinga2.log` log file in `/var/log/icinga2` will indicate whether this ApiListener [accepts config](13-distributed-monitoring-ha.md#zone-config-sync-permissions), or not. +Verify the object's [version](6-object-types.md#object-types) attribute on all nodes to +check whether the config update and reload was succesful or not. + ### Cluster Troubleshooting Overdue Check Results If your master does not receive check results (or any other events) from the child zones diff --git a/doc/18-language-reference.md b/doc/18-language-reference.md index 314c94935..fbdaa5ffd 100644 --- a/doc/18-language-reference.md +++ b/doc/18-language-reference.md @@ -43,7 +43,7 @@ Attribute | Description name | The name of the object. This attribute can be modified in the object definition to override the name specified with the `object` directive. type | The type of the object. -## Expressions +## Expressions The following expressions can be used on the right-hand side of assignments. diff --git a/doc/6-object-types.md b/doc/6-object-types.md index ea8ce9e03..46a7335bd 100644 --- a/doc/6-object-types.md +++ b/doc/6-object-types.md @@ -1,11 +1,26 @@ -# Object Types +# Config Object Types -This chapter provides an overview of all available object types which can be +This chapter provides an overview of all available config object types which can be instantiated using the `object` keyword. Additional details on configuration and runtime attributes and their description are explained as well. +Config objects share these runtime attributes which cannot be +modified by the user. You can access these attributes using +the [Icinga 2 API](9-icinga2-api.md#icinga2-api-config-objects). + + Name |Description + --------------------------|-------------------------- + version | Timestamp when the object was created or modified. Synced throughout cluster nodes. + type | Object type. + original_attributes | Original values of object attributes modified at runtime. + active | Object is active (e.g. a service being checked). + paused | Object has been paused at runtime (e.g. [IdoMysqlConnection](6-object-types.md#objecttype-idomysqlconnection). Defaults to `false`. + templates | Templates imported on object compilation. + package | [Configuration package name](9-icinga2-api.md#icinga2-api-config-management) this object belongs to. Local configuration is set to `_etc`, runtime created objects use `_api`. + + ## ApiListener ApiListener objects are used for distributed monitoring setups @@ -586,8 +601,8 @@ Runtime Attributes: last\_in\_downtime | Boolean | Whether the host was in a downtime when the last check occurred. acknowledgement | Number | The acknowledgement type (0 = NONE, 1 = NORMAL, 2 = STICKY). acknowledgement_expiry | Number | When the acknowledgement expires (as a UNIX timestamp; 0 = no expiry). - comments | Dictionary | The comments for this host. - downtimes | Dictionary | The downtimes for this host. + flapping_last_change | Number | When the last flapping change occurred (as a UNIX timestamp). + flapping | Boolean | Whether the host is flapping between states. state | Number | The current state (0 = UP, 1 = DOWN). last\_state | Number | The previous state (0 = UP, 1 = DOWN). last\_hard\_state | Number | The last hard state (0 = UP, 1 = DOWN). @@ -733,26 +748,26 @@ Data Categories: Name | Description | Required by ---------------------|------------------------|-------------------- - DbCatConfig | Configuration data | Icinga Web/Reporting - DbCatState | Current state data | Icinga Web/Reporting - DbCatAcknowledgement | Acknowledgements | Icinga Web/Reporting - DbCatComment | Comments | Icinga Web/Reporting - DbCatDowntime | Downtimes | Icinga Web/Reporting - DbCatEventHandler | Event handler data | Icinga Web/Reporting - DbCatExternalCommand | External commands | Icinga Web/Reporting - DbCatFlapping | Flap detection data | Icinga Web/Reporting + DbCatConfig | Configuration data | Icinga Web 2 + DbCatState | Current state data | Icinga Web 2 + DbCatAcknowledgement | Acknowledgements | Icinga Web 2 + DbCatComment | Comments | Icinga Web 2 + DbCatDowntime | Downtimes | Icinga Web 2 + DbCatEventHandler | Event handler data | Icinga Web 2 + DbCatExternalCommand | External commands | Icinga Web 2 + DbCatFlapping | Flap detection data | Icinga Web 2 DbCatCheck | Check results | -- - DbCatLog | Log messages | Icinga Web/Reporting - DbCatNotification | Notifications | Icinga Web/Reporting - DbCatProgramStatus | Program status data | Icinga Web/Reporting - DbCatRetention | Retention data | Icinga Web/Reporting - DbCatStateHistory | Historical state data | Icinga Web/Reporting + DbCatLog | Log messages | Icinga Web 2 + DbCatNotification | Notifications | Icinga Web 2 + DbCatProgramStatus | Program status data | Icinga Web 2 + DbCatRetention | Retention data | Icinga Web 2 + DbCatStateHistory | Historical state data | Icinga Web 2 Multiple categories can be combined using the `|` operator. In addition to the category flags listed above the `DbCatEverything` flag may be used as a shortcut for listing all flags. -External interfaces like Icinga Web require everything except `DbCatCheck` +External interfaces like Icinga Web 2 require everything except `DbCatCheck` which is the default value if `categories` is not set. ## IdoPgSqlConnection @@ -822,28 +837,29 @@ Data Categories: Name | Description | Required by ---------------------|------------------------|-------------------- - DbCatConfig | Configuration data | Icinga Web/Reporting - DbCatState | Current state data | Icinga Web/Reporting - DbCatAcknowledgement | Acknowledgements | Icinga Web/Reporting - DbCatComment | Comments | Icinga Web/Reporting - DbCatDowntime | Downtimes | Icinga Web/Reporting - DbCatEventHandler | Event handler data | Icinga Web/Reporting - DbCatExternalCommand | External commands | Icinga Web/Reporting - DbCatFlapping | Flap detection data | Icinga Web/Reporting + DbCatConfig | Configuration data | Icinga Web 2 + DbCatState | Current state data | Icinga Web 2 + DbCatAcknowledgement | Acknowledgements | Icinga Web 2 + DbCatComment | Comments | Icinga Web 2 + DbCatDowntime | Downtimes | Icinga Web 2 + DbCatEventHandler | Event handler data | Icinga Web 2 + DbCatExternalCommand | External commands | Icinga Web 2 + DbCatFlapping | Flap detection data | Icinga Web 2 DbCatCheck | Check results | -- - DbCatLog | Log messages | Icinga Web/Reporting - DbCatNotification | Notifications | Icinga Web/Reporting - DbCatProgramStatus | Program status data | Icinga Web/Reporting - DbCatRetention | Retention data | Icinga Web/Reporting - DbCatStateHistory | Historical state data | Icinga Web/Reporting + DbCatLog | Log messages | Icinga Web 2 + DbCatNotification | Notifications | Icinga Web 2 + DbCatProgramStatus | Program status data | Icinga Web 2 + DbCatRetention | Retention data | Icinga Web 2 + DbCatStateHistory | Historical state data | Icinga Web 2 Multiple categories can be combined using the `|` operator. In addition to the category flags listed above the `DbCatEverything` flag may be used as a shortcut for listing all flags. -External interfaces like Icinga Web require everything except `DbCatCheck` +External interfaces like Icinga Web 2 require everything except `DbCatCheck` which is the default value if `categories` is not set. + ## LiveStatusListener Livestatus API interface available as TCP or UNIX socket. Historical table queries @@ -1200,8 +1216,8 @@ Runtime Attributes: last\_in\_downtime | Boolean | Whether the service was in a downtime when the last check occurred. acknowledgement | Number | The acknowledgement type (0 = NONE, 1 = NORMAL, 2 = STICKY). acknowledgement_expiry | Number | When the acknowledgement expires (as a UNIX timestamp; 0 = no expiry). - comments | Dictionary | The comments for this service. - downtimes | Dictionary | The downtimes for this service. + flapping_last_change | Number | When the last flapping change occurred (as a UNIX timestamp). + flapping | Boolean | Whether the host is flapping between states. state | Number | The current state (0 = OK, 1 = WARNING, 2 = CRITICAL, 3 = UNKNOWN). last\_state | Number | The previous state (0 = OK, 1 = WARNING, 2 = CRITICAL, 3 = UNKNOWN). last\_hard\_state | Number | The last hard state (0 = OK, 1 = WARNING, 2 = CRITICAL, 3 = UNKNOWN). @@ -1439,3 +1455,53 @@ Configuration Attributes: endpoints |**Optional.** Dictionary with endpoints located in this zone. parent |**Optional.** The name of the parent zone. global |**Optional.** Whether configuration files for this zone should be synced to all endpoints. Defaults to false. + + + +# Value Types + +In addition to [expressions](18-language-reference.md#expressions) +used in object attribute assignments such as + +* [Numeric](18-language-reference.md#numeric-literals), [duration](8-language-reference.md#duration-literals), [string](18-language-reference.md#string-literals) and [boolean](18-language-reference.md#boolean-literals) literals +* [Array](18-language-reference.md#array) +* [Dictionary](18-language-reference.md#dictionary) + +Icinga 2 uses the following value types for runtime attributes +exposed via the [Icinga 2 API](9-icinga2-api.md#icinga2-api). + +## CheckResult + + Name | Type | Description + --------------------------|---------------|----------------- + exit_status | Number | The exit status returned by the check execution. + output | String | The check output. + performance_data | Array | Array of [performance data values](6-object-types.md#value-types-perfdatavalue). + check_source | String | Name of the node executing the check. + state | Number | The current state (0 = OK, 1 = WARNING, 2 = CRITICAL, 3 = UNKNOWN). + command | Value | Array of command with shell-escaped arguments or command line string. + execution_start | Number | Check execution start time (as a UNIX timestamp). + execution_end | Number | Check execution end time (as a UNIX timestamp). + schedule_start | Number | Scheduled check execution start time (as a UNIX timestamp). + schedule_end | Number | Scheduled check execution end time (as a UNIX timestamp). + active | Boolean | Whether the result is from an active or passive check. + vars_before | Dictionary | Internal attribute used for calculations. + vars_after | Dictionary | Internal attribute used for calculations. + +## PerfdataValue + +Icinga 2 parses performance data strings and exposes +the object to external interfaces (e.g. [GraphiteWriter](6-object-types.md#objecttype-graphitewriter) or the [Icinga 2 API](9-icinga2-api.md#icinga2-api)). + + Name | Type | Description + --------------------------|---------------|----------------- + label | String | Performance data label. + value | Number | Normalized performance data value without unit. + counter | Boolean | Enabled if the original value contains `c` as unit. Defaults to `false`. + unit | String | Unit of measurement (`seconds`, `bytes`. `percent`) according to the [plugin API](14-addons-plugins.md#plugin-api). + crit | Value | Critical threshold value. + warn | Value | Warning threshold value. + min | Value | Minimum value returned by the check. + max | Value | Maximum value returned by the check. + +