Add 'ttl' support for check result freshness via REST API

The `process-check-result` action can now optionally set the
`ttl` parameter. This overrules the configured freshness
check (check_interval).

The main idea behind this is to allow the external sender
to specify when the next check result is coming in.

For example, a backup script which should be run every
24h can specify the exact expected next check result.

The addition to the CheckResult class is necessary to
forward the check result throughout the cluster and
calculate the `next_check` value on each node. This
allows us to send in a check result on a satellite,
and the master determines the freshness and possible
notifications/state changes for Icinga Web 2.

doc/08-advanced-topics.md

@@ -378,6 +378,11 @@ the threshold is based on the last time a check result was received:
 
     (last check result time + check interval) > current time
 
+> **Tip**
+>
+> The [process-check-result](12-icinga2-api.md#icinga2-api-actions-process-check-result) REST API
+> action allows to overrule the pre-defined check interval with a specified TTL in Icinga 2 v2.9+.
+
 If the freshness checks fail, Icinga 2 will execute the defined check command.
 
 Best practice is to define a [dummy](10-icinga-template-library.md#itl-dummy) `check_command` which gets
@@ -1026,6 +1031,7 @@ to represent its internal state. The following types are exposed via the [API](1
   active                    | Boolean               | Whether the result is from an active or passive check.
   vars\_before              | Dictionary            | Internal attribute used for calculations.
   vars\_after               | Dictionary            | Internal attribute used for calculations.
+  ttl                       | Number                | Time-to-live duration in seconds for this check result. The next expected check result is `now + ttl` where freshness checks are executed.
 
 ### PerfdataValue <a id="advanced-value-types-perfdatavalue"></a>
 

doc/12-icinga2-api.md

@@ -811,6 +811,7 @@ Send a `POST` request to the URL endpoint `/v1/actions/process-check-result`.
   check\_source     | String       | **Optional.** Usually the name of the `command_endpoint`
   execution\_start  | Timestamp    | **Optional.** The timestamp where a script/process started its execution.
   execution\_end    | Timestamp    | **Optional.** The timestamp where a script/process ended its execution. This timestamp is used in features to determine e.g. the metric timestamp.
+  ttl               | Number       | **Optional.** Time-to-live duration in seconds for this check result. The next expected check result is `now + ttl` where freshness checks are executed.
 
 In addition to these parameters a [filter](12-icinga2-api.md#icinga2-api-filters) must be provided. The valid types for this action are `Host` and `Service`.
 

lib/icinga/apiactions.cpp

@@ -117,6 +117,10 @@ Dictionary::Ptr ApiActions::ProcessCheckResult(const ConfigObject::Ptr& object,
 	/* Mark this check result as passive. */
 	cr->SetActive(false);
 
+	/* Result TTL allows to overrule the next expected freshness check. */
+	if (params->Contains("ttl"))
+		cr->SetTtl(HttpUtility::GetLastParameter(params, "ttl"));
+
 	checkable->ProcessCheckResult(cr);
 
 	return ApiActions::CreateResult(200, "Successfully processed check result for object '" + checkable->GetName() + "'.");

lib/icinga/checkable-check.cpp

@@ -324,10 +324,18 @@ void Checkable::ProcessCheckResult(const CheckResult::Ptr& cr, const MessageOrig
 	if (cr->GetActive()) {
 		UpdateNextCheck(origin);
 	} else {
-		/* Reschedule the next check for passive check results. The side effect of
-		 * this is that for as long as we receive passive results for a service we
+		/* Reschedule the next check for external passive check results. The side effect of
+		 * this is that for as long as we receive results for a service we
 		 * won't execute any active checks. */
-		SetNextCheck(Utility::GetTime() + GetCheckInterval(), false, origin);
+		double offset;
+		double ttl = cr->GetTtl();
+
+		if (ttl > 0)
+			offset = ttl;
+		else
+			offset = GetCheckInterval();
+
+		SetNextCheck(Utility::GetTime() + offset, false, origin);
 	}
 
 	olock.Unlock();

lib/icinga/checkresult.ti

@@ -78,6 +78,7 @@ class CheckResult
 	};
 
 	[state] String check_source;
+	[state] double ttl;
 
 	[state] Dictionary::Ptr vars_before;
 	[state] Dictionary::Ptr vars_after;