Merge pull request #8040 from Icinga/feature/v1-actions-execute-command-8034

Add API endpoint: /v1/actions/execute-command
2025-07-29 00:24:23 +02:00 · 2020-12-02 10:53:24 +01:00 · 2020-12-02 10:53:24 +01:00 · bee4ac7f7c
commit bee4ac7f7c
parent 5224648ee0 7cf01008e6
46 changed files with 1520 additions and 197 deletions
--- a/doc/12-icinga2-api.md
+++ b/doc/12-icinga2-api.md
@ -1595,6 +1595,50 @@ $ curl -k -s -S -i -u root:icinga -H 'Accept: application/json' \
 }
 ```

+### execute-command <a id="icinga2-api-actions-execute-command"></a>
+
+Executes a particular check/notification/event-command on a particular
+endpoint in the context of a particular checkable. Example use cases:
+
+* Test a check command without actually triggering notifications
+* Reboot a node via an event command
+* Test a notification command without actually reproducing the notification reason
+
+Send a `POST` request to the URL endpoint `/v1/actions/execute-command`.
+
+  Parameter     | Type       | Description
+  --------------|------------|--------------
+  ttl           | Number     | **Required.** The time to live of the execution expressed in seconds.
+  command_type  | String     | **Optional.** The command type: `CheckCommand` or `EventCommand` or `NotificationCommand`. Default: `EventCommand`
+  command       | String     | **Optional.** The command to execute. Its type must the same as `command_type`. It can be a macro string. Default: depending on the `command_type` it's either `$check_command$`, `$event_command$` or `$notification_command$`   
+  endpoint      | String     | **Optional.** The endpoint to execute the command on. It can be a macro string. Default: `$command_endpoint$`.
+  macros        | Dictionary | **Optional.** Macro overrides. Default: `{}`
+  user          | String     | **Optional.** The user used for the notification command. 
+  notification  | String     | **Optional.** The notification used for the notification command.
+  
+Example:
+  
+```
+$ curl -k -s -S -i -u root:icinga -H 'Accept: application/json' \
+ -X POST 'https://localhost:5665/v1/actions/execute-command' \
+ -d '{"type": "Service", "service": "agent!custom_service", "ttl": 15, "macros": { "command_endpoint": "master", "ls_dir": "/tmp/foo" }, "command": "custom_command", "command_type": "CheckCommand" }'
+```
+
+```
+{
+    "results": [
+        {
+            "checkable": "agent!custom_service",
+            "code": 202.0,
+            "execution": "3541d906-9afe-4c0e-ae6d-f549ee9bb3e7",
+            "status": "Accepted"
+        }
+    ]
+}
+```
+
+You may poll the state of the execution by [querying](#icinga2-api-config-objects-query) the checkable's attribute `executions`.
+
 ## Event Streams <a id="icinga2-api-event-streams"></a>

 Event streams can be used to receive check results, downtimes, comments,
--- a/doc/19-technical-concepts.md
+++ b/doc/19-technical-concepts.md
@ -1286,7 +1286,10 @@ params    | Dictionary

 ##### Params

-Currently empty.
+Key                  | Type        | Description
+---------------------|-------------|------------------
+capabilities         | Number      | Bitmask, see `lib/remote/apilistener.hpp`.
+version              | Number      | Icinga 2 version, e.g. 21300 for v2.13.0.

 ##### Functions

@ -1817,11 +1820,14 @@ command\_type  | String        | `check_command` or `event_command`.
 command        | String        | CheckCommand or EventCommand name.
 check\_timeout | Number        | Check timeout of the checkable object, if specified as `check_timeout` attribute.
 macros         | Dictionary    | Command arguments as key/value pairs for remote execution.
+endpoint       | String        | The endpoint to execute the command on.
+deadline       | Number        | A Unix timestamp indicating the execution deadline
+source         | String        | The execution UUID


 ##### Functions

-**Event Sender:** This gets constructed directly in `Checkable::ExecuteCheck()` or `Checkable::ExecuteEventHandler()` when a remote command endpoint is configured.
+**Event Sender:** This gets constructed directly in `Checkable::ExecuteCheck()`, `Checkable::ExecuteEventHandler()` or `ApiActions::ExecuteCommand()` when a remote command endpoint is configured.

 * `Get{CheckCommand,EventCommand}()->Execute()` simulates an execution and extracts all command arguments into the `macro` dictionary (inside lib/methods tasks).
 * When the endpoint is connected, the message is constructed and sent directly.
@ -1831,6 +1837,7 @@ macros         | Dictionary    | Command arguments as key/value pairs for remote

 Special handling, calls `ClusterEvents::EnqueueCheck()` for command endpoint checks.
 This function enqueues check tasks into a queue which is controlled in `RemoteCheckThreadProc()`.
+If the `endpoint` parameter is specified and is not equal to the local endpoint then the message is forwarded to the correct endpoint zone. 

 ##### Permissions

@ -1852,6 +1859,78 @@ The returned messages are synced directly to the sender's endpoint, no cluster b

 > **Note**: EventCommand errors are just logged on the remote endpoint.

+### event::UpdateExecutions <a id="technical-concepts-json-rpc-messages-event-updateexecutions"></a>
+
+> Location: `clusterevents.cpp`
+
+##### Message Body
+
+Key       | Value
+----------|---------
+jsonrpc   | 2.0
+method    | event::UpdateExecutions
+params    | Dictionary
+
+##### Params
+
+Key            | Type          | Description
+---------------|---------------|------------------
+host           | String        | Host name.
+service        | String        | Service name.
+executions     | Dictionary    | Executions to be updated
+
+##### Functions
+
+**Event Sender:** `ClusterEvents::ExecutedCommandAPIHandler`, `ClusterEvents::UpdateExecutionsAPIHandler`, `ApiActions::ExecuteCommand`
+**Event Receiver:** `ClusterEvents::UpdateExecutionsAPIHandler` 
+
+##### Permissions
+
+The receiver will not process messages from not configured endpoints.
+
+Message updates will be dropped when:
+
+* Checkable does not exist.
+* Origin endpoint's zone is not allowed to access this checkable.
+
+### event::ExecutedCommand <a id="technical-concepts-json-rpc-messages-event-executedcommand"></a>
+
+> Location: `clusterevents.cpp`
+
+##### Message Body
+
+Key       | Value
+----------|---------
+jsonrpc   | 2.0
+method    | event::ExecutedCommand
+params    | Dictionary
+
+##### Params
+
+Key            | Type          | Description
+---------------|---------------|------------------
+host           | String        | Host name.
+service        | String        | Service name.
+execution      | String        | The execution ID executed.
+exitStatus     | Number        | The command exit status. 
+output         | String        | The command output.
+start          | Number        | The unix timestamp at the start of the command execution
+end            | Number        | The unix timestamp at the end of the command execution
+
+##### Functions
+
+**Event Sender:** `ClusterEvents::ExecuteCheckFromQueue`, `ClusterEvents::ExecuteCommandAPIHandler`
+**Event Receiver:** `ClusterEvents::ExecutedCommandAPIHandler` 
+
+##### Permissions
+
+The receiver will not process messages from not configured endpoints.
+
+Message updates will be dropped when:
+
+* Checkable does not exist.
+* Origin endpoint's zone is not allowed to access this checkable.
+
 #### config::Update <a id="technical-concepts-json-rpc-messages-config-update"></a>

 > Location: `apilistener-filesync.cpp`
--- a/lib/base/atomic.hpp
+++ b/lib/base/atomic.hpp
@ -4,6 +4,8 @@
 #define ATOMIC_H

 #include <atomic>
+#include <type_traits>
+#include <utility>

 namespace icinga
 {
@ -38,6 +40,80 @@ public:
 	}
 };

+/**
+ * Wraps T into a std::atomic<T>-like interface.
+ *
+ * @ingroup base
+ */
+template<class T>
+class NotAtomic
+{
+public:
+	inline T load() const
+	{
+		return m_Value;
+	}
+
+	inline void store(T desired)
+	{
+		m_Value = std::move(desired);
+	}
+
+	T m_Value;
+};
+
+/**
+ * Tells whether to use std::atomic<T> or NotAtomic<T>.
+ *
+ * @ingroup base
+ */
+template<class T>
+struct Atomicable
+{
+	// Doesn't work with too old compilers.
+	//static constexpr bool value = std::is_trivially_copyable<T>::value && sizeof(T) <= sizeof(void*);
+	static constexpr bool value = (std::is_fundamental<T>::value || std::is_pointer<T>::value) && sizeof(T) <= sizeof(void*);
+};
+
+/**
+ * Uses either std::atomic<T> or NotAtomic<T> depending on atomicable.
+ *
+ * @ingroup base
+ */
+template<bool atomicable>
+struct AtomicTemplate;
+
+template<>
+struct AtomicTemplate<false>
+{
+	template<class T>
+	struct tmplt
+	{
+		typedef NotAtomic<T> type;
+	};
+};
+
+template<>
+struct AtomicTemplate<true>
+{
+	template<class T>
+	struct tmplt
+	{
+		typedef std::atomic<T> type;
+	};
+};
+
+/**
+ * Uses either std::atomic<T> or NotAtomic<T> depending on T.
+ *
+ * @ingroup base
+ */
+template<class T>
+struct EventuallyAtomic
+{
+	typedef typename AtomicTemplate<Atomicable<T>::value>::template tmplt<T>::type type;
+};
+
 }

 #endif /* ATOMIC_H */
--- a/lib/base/configobject.ti
+++ b/lib/base/configobject.ti
@ -59,10 +59,10 @@ abstract class ConfigObject : ConfigObjectBase < ConfigType
 	[config, no_user_modify] String __name (Name);
 	[config, no_user_modify] String "name" (ShortName) {
 		get {{{
-			if (m_ShortName.IsEmpty())
+			if (m_ShortName.m_Value.IsEmpty())
 				return GetName();
 			else
-				return m_ShortName;
+				return m_ShortName.m_Value;
 		}}}
 	};
 	[config, no_user_modify] name(Zone) zone (ZoneName);
--- a/lib/db_ido/idochecktask.cpp
+++ b/lib/db_ido/idochecktask.cpp
@ -12,15 +12,39 @@
 #include "base/perfdatavalue.hpp"
 #include "base/configtype.hpp"
 #include "base/convert.hpp"
+#include <utility>

 using namespace icinga;

 REGISTER_FUNCTION_NONCONST(Internal, IdoCheck, &IdoCheckTask::ScriptFunc, "checkable:cr:resolvedMacros:useResolvedMacros");

+static void ReportIdoCheck(
+	const Checkable::Ptr& checkable, const CheckCommand::Ptr& commandObj,
+	const CheckResult::Ptr& cr, String output, ServiceState state = ServiceUnknown
+)
+{
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		double now = Utility::GetTime();
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = std::move(output);
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = state;
+
+		Checkable::ExecuteCommandProcessFinishedHandler(commandObj->GetName(), pr);
+	} else {
+		cr->SetState(state);
+		cr->SetOutput(output);
+		checkable->ProcessCheckResult(cr);
+	}
+}
+
 void IdoCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResult::Ptr& cr,
 	const Dictionary::Ptr& resolvedMacros, bool useResolvedMacros)
 {
-	CheckCommand::Ptr commandObj = checkable->GetCheckCommand();
+	ServiceState state;
+	CheckCommand::Ptr commandObj = CheckCommand::ExecuteOverride ? CheckCommand::ExecuteOverride : checkable->GetCheckCommand();
 	Value raw_command = commandObj->GetCommandLine();

 	Host::Ptr host;
@ -28,6 +52,10 @@ void IdoCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResult
 	tie(host, service) = GetHostService(checkable);

 	MacroProcessor::ResolverList resolvers;
+
+	if (MacroResolver::OverrideMacros)
+		resolvers.emplace_back("override", MacroResolver::OverrideMacros);
+
 	if (service)
 		resolvers.emplace_back("service", service);
 	resolvers.emplace_back("host", host);
@ -61,25 +89,19 @@ void IdoCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResult
 		return;

 	if (idoType.IsEmpty()) {
-		cr->SetOutput("Attribute 'ido_type' must be set.");
-		cr->SetState(ServiceUnknown);
-		checkable->ProcessCheckResult(cr);
+		ReportIdoCheck(checkable, commandObj, cr, "Attribute 'ido_type' must be set.");
 		return;
 	}

 	if (idoName.IsEmpty()) {
-		cr->SetOutput("Attribute 'ido_name' must be set.");
-		cr->SetState(ServiceUnknown);
-		checkable->ProcessCheckResult(cr);
+		ReportIdoCheck(checkable, commandObj, cr, "Attribute 'ido_name' must be set.");
 		return;
 	}

 	Type::Ptr type = Type::GetByName(idoType);

 	if (!type || !DbConnection::TypeInstance->IsAssignableFrom(type)) {
-		cr->SetOutput("DB IDO type '" + idoType + "' is invalid.");
-		cr->SetState(ServiceUnknown);
-		checkable->ProcessCheckResult(cr);
+		ReportIdoCheck(checkable, commandObj, cr, "DB IDO type '" + idoType + "' is invalid.");
 		return;
 	}

@ -89,18 +111,14 @@ void IdoCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResult
 	DbConnection::Ptr conn = static_pointer_cast<DbConnection>(dtype->GetObject(idoName));

 	if (!conn) {
-		cr->SetOutput("DB IDO connection '" + idoName + "' does not exist.");
-		cr->SetState(ServiceUnknown);
-		checkable->ProcessCheckResult(cr);
+		ReportIdoCheck(checkable, commandObj, cr, "DB IDO connection '" + idoName + "' does not exist.");
 		return;
 	}

 	double qps = conn->GetQueryCount(60) / 60.0;

 	if (conn->IsPaused()) {
-		cr->SetOutput("DB IDO connection is temporarily disabled on this cluster instance.");
-		cr->SetState(ServiceOK);
-		checkable->ProcessCheckResult(cr);
+		ReportIdoCheck(checkable, commandObj, cr, "DB IDO connection is temporarily disabled on this cluster instance.", ServiceOK);
 		return;
 	}

@ -108,15 +126,13 @@ void IdoCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResult

 	if (!conn->GetConnected()) {
 		if (conn->GetShouldConnect()) {
-			cr->SetOutput("Could not connect to the database server.");
-			cr->SetState(ServiceCritical);
+			ReportIdoCheck(checkable, commandObj, cr, "Could not connect to the database server.", ServiceCritical);
 		} else {
-			cr->SetOutput("Not currently enabled: Another cluster instance is responsible for the IDO database.");
-			cr->SetState(ServiceOK);
+			ReportIdoCheck(
+				checkable, commandObj, cr,
+				"Not currently enabled: Another cluster instance is responsible for the IDO database.", ServiceOK
+			);
 		}
-
-		checkable->ProcessCheckResult(cr);
-
 		return;
 	}

@ -130,13 +146,13 @@ void IdoCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResult
 		    << " Queries per second: " << std::fixed << std::setprecision(3) << qps
 		    << " Pending queries: " << std::fixed << std::setprecision(3) << pendingQueries << ".";

-		cr->SetState(ServiceWarning);
+		state = ServiceWarning;
 	} else {
 		msgbuf << "Connected to the database server (Schema version: '" << schema_version << "')."
 		    << " Queries per second: " << std::fixed << std::setprecision(3) << qps
 		    << " Pending queries: " << std::fixed << std::setprecision(3) << pendingQueries << ".";

-		cr->SetState(ServiceOK);
+		state = ServiceOK;
 	}

 	if (conn->GetEnableHa()) {
@ -149,27 +165,25 @@ void IdoCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResult
 	if (missingQueriesCritical.IsEmpty() && qps < queriesCritical) {
 		msgbuf << " " << qps << " queries/s lower than critical threshold (" << queriesCritical << " queries/s).";

-		cr->SetState(ServiceCritical);
+		state= ServiceCritical;
 	} else if (missingQueriesWarning.IsEmpty() && qps < queriesWarning) {
 		msgbuf << " " << qps << " queries/s lower than warning threshold (" << queriesWarning << " queries/s).";

-		cr->SetState(ServiceWarning);
+		state = ServiceWarning;
 	}

 	if (missingPendingQueriesCritical.IsEmpty() && pendingQueries > pendingQueriesCritical) {
 		msgbuf << " " << pendingQueries << " pending queries greater than critical threshold ("
 		    << pendingQueriesCritical << " queries).";

-		cr->SetState(ServiceCritical);
+		state = ServiceCritical;
 	} else if (missingPendingQueriesWarning.IsEmpty() && pendingQueries > pendingQueriesWarning) {
 		msgbuf << " " << pendingQueries << " pending queries greater than warning threshold ("
 		    << pendingQueriesWarning << " queries).";

-		cr->SetState(ServiceWarning);
+		state = ServiceWarning;
 	}

-	cr->SetOutput(msgbuf.str());
-
 	cr->SetPerformanceData(new Array({
 		{ new PerfdataValue("queries", qps, false, "", queriesWarning, queriesCritical) },
 		{ new PerfdataValue("queries_1min", conn->GetQueryCount(60)) },
@ -178,5 +192,5 @@ void IdoCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResult
 		{ new PerfdataValue("pending_queries", pendingQueries, false, "", pendingQueriesWarning, pendingQueriesCritical) }
 	}));

-	checkable->ProcessCheckResult(cr);
+	ReportIdoCheck(checkable, commandObj, cr, msgbuf.str(), state);
 }
--- a/lib/icinga/apiactions.cpp
+++ b/lib/icinga/apiactions.cpp
@ -8,12 +8,16 @@
 #include "icinga/checkcommand.hpp"
 #include "icinga/eventcommand.hpp"
 #include "icinga/notificationcommand.hpp"
+#include "icinga/clusterevents.hpp"
 #include "remote/apiaction.hpp"
 #include "remote/apilistener.hpp"
+#include "remote/filterutility.hpp"
 #include "remote/pkiutility.hpp"
 #include "remote/httputility.hpp"
 #include "base/utility.hpp"
 #include "base/convert.hpp"
+#include "base/defer.hpp"
+#include "remote/actionshandler.hpp"
 #include <fstream>

 using namespace icinga;
@ -31,6 +35,7 @@ REGISTER_APIACTION(remove_downtime, "Service;Host;Downtime", &ApiActions::Remove
 REGISTER_APIACTION(shutdown_process, "", &ApiActions::ShutdownProcess);
 REGISTER_APIACTION(restart_process, "", &ApiActions::RestartProcess);
 REGISTER_APIACTION(generate_ticket, "", &ApiActions::GenerateTicket);
+REGISTER_APIACTION(execute_command, "Service;Host", &ApiActions::ExecuteCommand);

 Dictionary::Ptr ApiActions::CreateResult(int code, const String& status,
 	const Dictionary::Ptr& additional)
@ -550,3 +555,328 @@ Dictionary::Ptr ApiActions::GenerateTicket(const ConfigObject::Ptr&,
 	return ApiActions::CreateResult(200, "Generated PKI ticket '" + ticket + "' for common name '"
 		+ cn + "'.", additional);
 }
+
+Value ApiActions::GetSingleObjectByNameUsingPermissions(const String& type, const String& objectName, const ApiUser::Ptr& user)
+{
+	Dictionary::Ptr queryParams = new Dictionary();
+	queryParams->Set("type", type);
+	queryParams->Set(type.ToLower(), objectName);
+
+	QueryDescription qd;
+	qd.Types.insert(type);
+	qd.Permission = "objects/query/" + type;
+
+	std::vector<Value> objs;
+
+	try {
+		objs = FilterUtility::GetFilterTargets(qd, queryParams, user);
+	} catch (const std::exception& ex) {
+		Log(LogWarning, "ApiActions") << DiagnosticInformation(ex);
+		return nullptr;
+	}
+
+	if (objs.empty())
+		return nullptr;
+
+	return objs.at(0);
+};
+
+Dictionary::Ptr ApiActions::ExecuteCommand(const ConfigObject::Ptr& object, const Dictionary::Ptr& params)
+{
+	ApiListener::Ptr listener = ApiListener::GetInstance();
+
+	if (!listener)
+		BOOST_THROW_EXCEPTION(std::invalid_argument("No ApiListener instance configured."));
+
+	/* Get command_type */
+	String command_type = "EventCommand";
+
+	if (params->Contains("command_type"))
+		command_type = HttpUtility::GetLastParameter(params, "command_type");
+
+	/* Validate command_type */
+	if (command_type != "EventCommand" && command_type != "CheckCommand" && command_type != "NotificationCommand")
+		return ApiActions::CreateResult(400, "Invalid command_type '" + command_type + "'.");
+
+	Checkable::Ptr checkable = dynamic_pointer_cast<Checkable>(object);
+
+	if (!checkable)
+		return ApiActions::CreateResult(404, "Can't start a command execution for a non-existent object.");
+
+	/* Get TTL param */
+	if (!params->Contains("ttl"))
+		return ApiActions::CreateResult(400, "Parameter ttl is required.");
+
+	double ttl = HttpUtility::GetLastParameter(params, "ttl");
+
+	if (ttl <= 0)
+		return ApiActions::CreateResult(400, "Parameter ttl must be greater than 0.");
+
+	ObjectLock oLock (checkable);
+
+	Host::Ptr host;
+	Service::Ptr service;
+	tie(host, service) = GetHostService(checkable);
+
+	String endpoint = "$command_endpoint$";
+
+	if (params->Contains("endpoint"))
+		endpoint = HttpUtility::GetLastParameter(params, "endpoint");
+
+	MacroProcessor::ResolverList resolvers;
+	Value macros;
+
+	if (params->Contains("macros")) {
+		macros = HttpUtility::GetLastParameter(params, "macros");
+		if (macros.IsObjectType<Dictionary>()) {
+			resolvers.emplace_back("override", macros);
+		} else {
+			return ApiActions::CreateResult(400, "Parameter macros must be a dictionary.");
+		}
+	}
+
+	if (service)
+		resolvers.emplace_back("service", service);
+
+	resolvers.emplace_back("host", host);
+	resolvers.emplace_back("icinga", IcingaApplication::GetInstance());
+
+	String resolved_endpoint = MacroProcessor::ResolveMacros(
+		endpoint, resolvers, checkable->GetLastCheckResult(),
+		nullptr, MacroProcessor::EscapeCallback(), nullptr, false
+	);
+
+	if (!ActionsHandler::AuthenticatedApiUser)
+		BOOST_THROW_EXCEPTION(std::invalid_argument("Can't find API user."));
+
+	/* Get endpoint */
+	Endpoint::Ptr endpointPtr = GetSingleObjectByNameUsingPermissions(Endpoint::GetTypeName(), resolved_endpoint, ActionsHandler::AuthenticatedApiUser);
+
+	if (!endpointPtr)
+		return ApiActions::CreateResult(404, "Can't find a valid endpoint for '" + resolved_endpoint + "'.");
+
+	/* Get command */
+	String command;
+
+	if (!params->Contains("command")) {
+		if (command_type == "CheckCommand" ) {
+			command = "$check_command$";
+		} else if (command_type == "EventCommand") {
+			command = "$event_command$";
+		} else if (command_type == "NotificationCommand") {
+			command = "$notification_command$";
+		}
+	} else {
+		command = HttpUtility::GetLastParameter(params, "command");
+	}
+
+	/* Resolve command macro */
+	String resolved_command = MacroProcessor::ResolveMacros(
+		command, resolvers, checkable->GetLastCheckResult(), nullptr,
+		MacroProcessor::EscapeCallback(), nullptr, false
+	);
+
+	CheckResult::Ptr cr = checkable->GetLastCheckResult();
+
+	if (!cr)
+		cr = new CheckResult();
+
+	/* Check if resolved_command exists and it is of type command_type */
+	Dictionary::Ptr execMacros = new Dictionary();
+
+	MacroResolver::OverrideMacros = macros;
+	Defer o ([]() {
+		MacroResolver::OverrideMacros = nullptr;
+	});
+
+	/* Create execution parameters */
+	Dictionary::Ptr execParams = new Dictionary();
+
+	if (command_type == "CheckCommand") {
+		CheckCommand::Ptr cmd = GetSingleObjectByNameUsingPermissions(CheckCommand::GetTypeName(), resolved_command, ActionsHandler::AuthenticatedApiUser);
+
+		if (!cmd)
+			return ApiActions::CreateResult(404, "Can't find a valid " + command_type + " for '" + resolved_command + "'.");
+		else {
+			CheckCommand::ExecuteOverride = cmd;
+			Defer resetCheckCommandOverride([]() {
+				CheckCommand::ExecuteOverride = nullptr;
+			});
+			cmd->Execute(checkable, cr, execMacros, false);
+		}
+	} else if (command_type == "EventCommand") {
+		EventCommand::Ptr cmd = GetSingleObjectByNameUsingPermissions(EventCommand::GetTypeName(), resolved_command, ActionsHandler::AuthenticatedApiUser);
+
+		if (!cmd)
+			return ApiActions::CreateResult(404, "Can't find a valid " + command_type + " for '" + resolved_command + "'.");
+		else {
+			EventCommand::ExecuteOverride = cmd;
+			Defer resetEventCommandOverride ([]() {
+				EventCommand::ExecuteOverride = nullptr;
+			});
+			cmd->Execute(checkable, execMacros, false);
+		}
+	} else if (command_type == "NotificationCommand") {
+		NotificationCommand::Ptr cmd = GetSingleObjectByNameUsingPermissions(NotificationCommand::GetTypeName(), resolved_command, ActionsHandler::AuthenticatedApiUser);
+
+		if (!cmd)
+			return ApiActions::CreateResult(404, "Can't find a valid " + command_type + " for '" + resolved_command + "'.");
+		else {
+			/* Get user */
+			String user_string = "";
+
+			if (params->Contains("user"))
+				user_string = HttpUtility::GetLastParameter(params, "user");
+
+			/* Resolve user macro */
+			String resolved_user = MacroProcessor::ResolveMacros(
+				user_string, resolvers, checkable->GetLastCheckResult(), nullptr,
+				MacroProcessor::EscapeCallback(), nullptr, false
+			);
+
+			User::Ptr user = GetSingleObjectByNameUsingPermissions(User::GetTypeName(), resolved_user, ActionsHandler::AuthenticatedApiUser);
+
+			if (!user)
+				return ApiActions::CreateResult(404, "Can't find a valid user for '" + resolved_user + "'.");
+
+			execParams->Set("user", user->GetName());
+
+			/* Get notification */
+			String notification_string = "";
+
+			if (params->Contains("notification"))
+				notification_string = HttpUtility::GetLastParameter(params, "notification");
+
+			/* Resolve notification macro */
+			String resolved_notification = MacroProcessor::ResolveMacros(
+				notification_string, resolvers, checkable->GetLastCheckResult(), nullptr,
+				MacroProcessor::EscapeCallback(), nullptr, false
+			);
+
+			Notification::Ptr notification = GetSingleObjectByNameUsingPermissions(Notification::GetTypeName(), resolved_notification, ActionsHandler::AuthenticatedApiUser);
+
+			if (!notification)
+				return ApiActions::CreateResult(404, "Can't find a valid notification for '" + resolved_notification + "'.");
+
+			execParams->Set("notification", notification->GetName());
+
+			NotificationCommand::ExecuteOverride = cmd;
+			Defer resetNotificationCommandOverride ([]() {
+				NotificationCommand::ExecuteOverride = nullptr;
+			});
+
+			cmd->Execute(notification, user, cr, NotificationType::NotificationCustom,
+				ActionsHandler::AuthenticatedApiUser->GetName(), "", execMacros, false);
+		}
+	}
+
+	/* This generates a UUID */
+	String uuid = Utility::NewUniqueID();
+
+	/* Create the deadline */
+	double deadline = Utility::GetTime() + ttl;
+
+	/* Update executions */
+	Dictionary::Ptr pending_execution = new Dictionary();
+	pending_execution->Set("pending", true);
+	pending_execution->Set("deadline", deadline);
+	Dictionary::Ptr executions = checkable->GetExecutions();
+
+	if (!executions)
+		executions = new Dictionary();
+
+	executions->Set(uuid, pending_execution);
+	checkable->SetExecutions(executions);
+
+	/* Broadcast the update */
+	Dictionary::Ptr executionsToBroadcast = new Dictionary();
+	executionsToBroadcast->Set(uuid, pending_execution);
+	Dictionary::Ptr updateParams = new Dictionary();
+	updateParams->Set("host", host->GetName());
+
+	if (service)
+		updateParams->Set("service", service->GetShortName());
+
+	updateParams->Set("executions", executionsToBroadcast);
+
+	Dictionary::Ptr updateMessage = new Dictionary();
+	updateMessage->Set("jsonrpc", "2.0");
+	updateMessage->Set("method", "event::UpdateExecutions");
+	updateMessage->Set("params", updateParams);
+
+	MessageOrigin::Ptr origin = new MessageOrigin();
+	listener->RelayMessage(origin, checkable, updateMessage, true);
+
+	/* Populate execution parameters */
+	if (command_type == "CheckCommand")
+		execParams->Set("command_type", "check_command");
+	else if (command_type == "EventCommand")
+		execParams->Set("command_type", "event_command");
+	else if (command_type == "NotificationCommand")
+		execParams->Set("command_type", "notification_command");
+
+	execParams->Set("command", resolved_command);
+	execParams->Set("host", host->GetName());
+
+	if (service)
+		execParams->Set("service", service->GetShortName());
+
+	/*
+	 * If the host/service object specifies the 'check_timeout' attribute,
+	 * forward this to the remote endpoint to limit the command execution time.
+	 */
+	if (!checkable->GetCheckTimeout().IsEmpty())
+		execParams->Set("check_timeout", checkable->GetCheckTimeout());
+
+	execParams->Set("source", uuid);
+	execParams->Set("deadline", deadline);
+	execParams->Set("macros", execMacros);
+	execParams->Set("endpoint", resolved_endpoint);
+
+	/* Execute command */
+	bool local = endpointPtr == Endpoint::GetLocalEndpoint();
+
+	if (local) {
+		ClusterEvents::ExecuteCommandAPIHandler(origin, execParams);
+	} else {
+		/* Check if the child endpoints have Icinga version >= 2.13 */
+		Zone::Ptr localZone = Zone::GetLocalZone();
+		for (const Zone::Ptr& zone : ConfigType::GetObjectsByType<Zone>()) {
+			/* Fetch immediate child zone members */
+			if (zone->GetParent() == localZone && zone->CanAccessObject(endpointPtr->GetZone())) {
+				std::set<Endpoint::Ptr> endpoints = zone->GetEndpoints();
+
+				for (const Endpoint::Ptr& childEndpoint : endpoints) {
+					if (!(childEndpoint->GetCapabilities() & (uint_fast64_t)ApiCapabilities::ExecuteArbitraryCommand)) {
+						/* Update execution */
+						double now = Utility::GetTime();
+						pending_execution->Set("exit", 126);
+						pending_execution->Set("output", "Endpoint '" + childEndpoint->GetName() + "' doesn't support executing arbitrary commands.");
+						pending_execution->Set("start", now);
+						pending_execution->Set("end", now);
+						pending_execution->Remove("pending");
+
+						listener->RelayMessage(origin, checkable, updateMessage, true);
+
+						Dictionary::Ptr result = new Dictionary();
+						result->Set("checkable", checkable->GetName());
+						result->Set("execution", uuid);
+						return ApiActions::CreateResult(202, "Accepted", result);
+					}
+				}
+			}
+		}
+
+		Dictionary::Ptr execMessage = new Dictionary();
+		execMessage->Set("jsonrpc", "2.0");
+		execMessage->Set("method", "event::ExecuteCommand");
+		execMessage->Set("params", execParams);
+
+		listener->RelayMessage(origin, endpointPtr->GetZone(), execMessage, true);
+	}
+
+	Dictionary::Ptr result = new Dictionary();
+	result->Set("checkable", checkable->GetName());
+	result->Set("execution", uuid);
+	return ApiActions::CreateResult(202, "Accepted", result);
+}
--- a/lib/icinga/apiactions.hpp
+++ b/lib/icinga/apiactions.hpp
@ -6,6 +6,7 @@
 #include "icinga/i2-icinga.hpp"
 #include "base/configobject.hpp"
 #include "base/dictionary.hpp"
+#include "remote/apiuser.hpp"

 namespace icinga
 {
@ -29,9 +30,11 @@ public:
 	static Dictionary::Ptr ShutdownProcess(const ConfigObject::Ptr& object, const Dictionary::Ptr& params);
 	static Dictionary::Ptr RestartProcess(const ConfigObject::Ptr& object, const Dictionary::Ptr& params);
 	static Dictionary::Ptr GenerateTicket(const ConfigObject::Ptr& object, const Dictionary::Ptr& params);
+	static Dictionary::Ptr ExecuteCommand(const ConfigObject::Ptr& object, const Dictionary::Ptr& params);

 private:
 	static Dictionary::Ptr CreateResult(int code, const String& status, const Dictionary::Ptr& additional = nullptr);
+	static Value GetSingleObjectByNameUsingPermissions(const String& type, const String& value, const ApiUser::Ptr& user);
 };

 }
--- a/lib/icinga/checkable.cpp
+++ b/lib/icinga/checkable.cpp
@ -20,6 +20,9 @@ boost::signals2::signal<void (const Checkable::Ptr&, const String&, double, cons
 boost::signals2::signal<void (const Checkable::Ptr&, double)> Checkable::OnFlappingChange;

 static Timer::Ptr l_CheckablesFireSuppressedNotifications;
+static Timer::Ptr l_CleanDeadlinedExecutions;
+
+thread_local std::function<void(const Value& commandLine, const ProcessResult&)> Checkable::ExecuteCommandProcessFinishedHandler;

 void Checkable::StaticInitialize()
 {
@ -86,6 +89,11 @@ void Checkable::Start(bool runtimeCreated)
 		l_CheckablesFireSuppressedNotifications->SetInterval(5);
 		l_CheckablesFireSuppressedNotifications->OnTimerExpired.connect(&Checkable::FireSuppressedNotifications);
 		l_CheckablesFireSuppressedNotifications->Start();
+
+		l_CleanDeadlinedExecutions = new Timer();
+		l_CleanDeadlinedExecutions->SetInterval(300);
+		l_CleanDeadlinedExecutions->OnTimerExpired.connect(&Checkable::CleanDeadlinedExecutions);
+		l_CleanDeadlinedExecutions->Start();
 	});
 }

@ -265,3 +273,34 @@ void Checkable::ValidateMaxCheckAttempts(const Lazy<int>& lvalue, const Validati
 	if (lvalue() <= 0)
 		BOOST_THROW_EXCEPTION(ValidationError(this, { "max_check_attempts" }, "Value must be greater than 0."));
 }
+
+void Checkable::CleanDeadlinedExecutions(const Timer * const&)
+{
+	double now = Utility::GetTime();
+	Dictionary::Ptr executions;
+	Dictionary::Ptr execution;
+
+	for (auto& host : ConfigType::GetObjectsByType<Host>()) {
+		executions = host->GetExecutions();
+		if (executions) {
+			for (const String& key : executions->GetKeys()) {
+				execution = executions->Get(key);
+				if (execution->Contains("deadline") && now > execution->Get("deadline")) {
+					executions->Remove(key);
+				}
+			}
+		}
+	}
+
+	for (auto& service : ConfigType::GetObjectsByType<Service>()) {
+		executions = service->GetExecutions();
+		if (executions) {
+			for (const String& key : executions->GetKeys()) {
+				execution = executions->Get(key);
+				if (execution->Contains("deadline") && now > execution->Get("deadline")) {
+					executions->Remove(key);
+				}
+			}
+		}
+	}
+}
--- a/lib/icinga/checkable.hpp
+++ b/lib/icinga/checkable.hpp
@ -5,6 +5,7 @@

 #include "base/atomic.hpp"
 #include "base/timer.hpp"
+#include "base/process.hpp"
 #include "icinga/i2-icinga.hpp"
 #include "icinga/checkable-ti.hpp"
 #include "icinga/timeperiod.hpp"
@ -14,6 +15,7 @@
 #include "remote/endpoint.hpp"
 #include "remote/messageorigin.hpp"
 #include <cstdint>
+#include <functional>

 namespace icinga
 {
@ -55,6 +57,7 @@ public:
 	DECLARE_OBJECTNAME(Checkable);

 	static void StaticInitialize();
+	static thread_local std::function<void(const Value& commandLine, const ProcessResult&)> ExecuteCommandProcessFinishedHandler;

 	Checkable();

@ -205,6 +208,7 @@ private:
 	static void NotifyDowntimeEnd(const Downtime::Ptr& downtime);

 	static void FireSuppressedNotifications(const Timer * const&);
+	static void CleanDeadlinedExecutions(const Timer * const&);

 	/* Comments */
 	std::set<Comment::Ptr> m_Comments;
--- a/lib/icinga/checkable.ti
+++ b/lib/icinga/checkable.ti
@ -175,6 +175,9 @@ abstract class Checkable : CustomVarObject
 			return Endpoint::GetByName(GetCommandEndpointRaw());
 		}}}
 	};
+
+	[state, no_user_modify] Dictionary::Ptr executions;
+	[state, no_user_view, no_user_modify] Dictionary::Ptr pending_executions;
 };

 }
--- a/lib/icinga/checkcommand.cpp
+++ b/lib/icinga/checkcommand.cpp
@ -8,6 +8,8 @@ using namespace icinga;

 REGISTER_TYPE(CheckCommand);

+thread_local CheckCommand::Ptr CheckCommand::ExecuteOverride;
+
 void CheckCommand::Execute(const Checkable::Ptr& checkable, const CheckResult::Ptr& cr,
 	const Dictionary::Ptr& resolvedMacros, bool useResolvedMacros)
 {
--- a/lib/icinga/checkcommand.hpp
+++ b/lib/icinga/checkcommand.hpp
@ -20,6 +20,8 @@ public:
 	DECLARE_OBJECT(CheckCommand);
 	DECLARE_OBJECTNAME(CheckCommand);

+	static thread_local CheckCommand::Ptr ExecuteOverride;
+
 	virtual void Execute(const Checkable::Ptr& checkable, const CheckResult::Ptr& cr,
 		const Dictionary::Ptr& resolvedMacros = nullptr,
 		bool useResolvedMacros = false);
--- a/lib/icinga/clusterevents-check.cpp
+++ b/lib/icinga/clusterevents-check.cpp
@ -4,6 +4,7 @@
 #include "icinga/icingaapplication.hpp"
 #include "remote/apilistener.hpp"
 #include "base/configuration.hpp"
+#include "base/defer.hpp"
 #include "base/serializer.hpp"
 #include "base/exception.hpp"
 #include <boost/thread/once.hpp>
@ -75,44 +76,127 @@ void ClusterEvents::EnqueueCheck(const MessageOrigin::Ptr& origin, const Diction
 	}
 }

+static void SendEventExecutedCommand(const Dictionary::Ptr& params, long exitStatus, const String& output,
+	double start, double end, const ApiListener::Ptr& listener, const MessageOrigin::Ptr& origin,
+	const Endpoint::Ptr& sourceEndpoint)
+{
+	Dictionary::Ptr executedParams = new Dictionary();
+	executedParams->Set("execution", params->Get("source"));
+	executedParams->Set("host", params->Get("host"));
+
+	if (params->Contains("service"))
+		executedParams->Set("service", params->Get("service"));
+
+	executedParams->Set("exit", exitStatus);
+	executedParams->Set("output", output);
+	executedParams->Set("start", start);
+	executedParams->Set("end", end);
+
+	if (origin->IsLocal()) {
+		ClusterEvents::ExecutedCommandAPIHandler(origin, executedParams);
+	} else {
+		Dictionary::Ptr executedMessage = new Dictionary();
+		executedMessage->Set("jsonrpc", "2.0");
+		executedMessage->Set("method", "event::ExecutedCommand");
+		executedMessage->Set("params", executedParams);
+
+		listener->SyncSendMessage(sourceEndpoint, executedMessage);
+	}
+}
+
 void ClusterEvents::ExecuteCheckFromQueue(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params) {

-	Endpoint::Ptr sourceEndpoint = origin->FromClient->GetEndpoint();
+	Endpoint::Ptr sourceEndpoint;
+
+	if (origin->FromClient) {
+		sourceEndpoint = origin->FromClient->GetEndpoint();
+	} else if (origin->IsLocal()){
+		sourceEndpoint = Endpoint::GetLocalEndpoint();
+	}

 	if (!sourceEndpoint || (origin->FromZone && !Zone::GetLocalZone()->IsChildOf(origin->FromZone))) {
 		Log(LogNotice, "ClusterEvents")
-				<< "Discarding 'execute command' message from '" << origin->FromClient->GetIdentity() << "': Invalid endpoint origin (client not allowed).";
+			<< "Discarding 'execute command' message from '" << origin->FromClient->GetIdentity() << "': Invalid endpoint origin (client not allowed).";
 		return;
 	}

 	ApiListener::Ptr listener = ApiListener::GetInstance();

 	if (!listener) {
-		Log(LogCritical, "ApiListener", "No instance available.");
+		Log(LogCritical, "ApiListener") << "No instance available.";
 		return;
 	}

-	if (!listener->GetAcceptCommands()) {
-		Log(LogWarning, "ApiListener")
-				<< "Ignoring command. '" << listener->GetName() << "' does not accept commands.";
+	Defer resetExecuteCommandProcessFinishedHandler ([]() {
+		Checkable::ExecuteCommandProcessFinishedHandler = nullptr;
+	});

-		Host::Ptr host = new Host();
-		Dictionary::Ptr attrs = new Dictionary();
+	if (params->Contains("source")) {
+		String uuid = params->Get("source");

-		attrs->Set("__name", params->Get("host"));
-		attrs->Set("type", "Host");
-		attrs->Set("enable_active_checks", false);
-
-		Deserialize(host, attrs, false, FAConfig);
+		String checkableName = params->Get("host");

 		if (params->Contains("service"))
-			host->SetExtension("agent_service_name", params->Get("service"));
+			checkableName += "!" + params->Get("service");

-		CheckResult::Ptr cr = new CheckResult();
-		cr->SetState(ServiceUnknown);
-		cr->SetOutput("Endpoint '" + Endpoint::GetLocalEndpoint()->GetName() + "' does not accept commands.");
-		Dictionary::Ptr message = MakeCheckResultMessage(host, cr);
-		listener->SyncSendMessage(sourceEndpoint, message);
+		/* Check deadline */
+		double deadline = params->Get("deadline");
+
+		if (Utility::GetTime() > deadline) {
+			Log(LogNotice, "ApiListener")
+				<< "Discarding 'ExecuteCheckFromQueue' event for checkable '" << checkableName
+				<< "' from '" << origin->FromClient->GetIdentity() << "': Deadline has expired.";
+			return;
+		}
+
+		Checkable::ExecuteCommandProcessFinishedHandler = [checkableName, listener, sourceEndpoint, origin, params] (const Value& commandLine, const ProcessResult& pr) {
+			if (params->Get("command_type") == "check_command") {
+				Checkable::CurrentConcurrentChecks.fetch_sub(1);
+				Checkable::DecreasePendingChecks();
+			}
+
+			if (pr.ExitStatus > 3) {
+				Process::Arguments parguments = Process::PrepareCommand(commandLine);
+				Log(LogWarning, "ApiListener")
+					<< "Command for object '" << checkableName << "' (PID: " << pr.PID
+					<< ", arguments: " << Process::PrettyPrintArguments(parguments) << ") terminated with exit code "
+					<< pr.ExitStatus << ", output: " << pr.Output;
+			}
+
+			SendEventExecutedCommand(params, pr.ExitStatus, pr.Output, pr.ExecutionStart, pr.ExecutionEnd, listener,
+									origin, sourceEndpoint);
+		};
+	}
+
+	if (!listener->GetAcceptCommands() && !origin->IsLocal()) {
+		Log(LogWarning, "ApiListener")
+			<< "Ignoring command. '" << listener->GetName() << "' does not accept commands.";
+
+		String output = "Endpoint '" + Endpoint::GetLocalEndpoint()->GetName() + "' does not accept commands.";
+
+		if (params->Contains("source")) {
+			double now = Utility::GetTime();
+			SendEventExecutedCommand(params, 126, output, now, now, listener, origin, sourceEndpoint);
+		} else {
+			Host::Ptr host = new Host();
+			Dictionary::Ptr attrs = new Dictionary();
+
+			attrs->Set("__name", params->Get("host"));
+			attrs->Set("type", "Host");
+			attrs->Set("enable_active_checks", false);
+
+			Deserialize(host, attrs, false, FAConfig);
+
+			if (params->Contains("service"))
+				host->SetExtension("agent_service_name", params->Get("service"));
+
+			CheckResult::Ptr cr = new CheckResult();
+			cr->SetState(ServiceUnknown);
+			cr->SetOutput(output);
+
+			Dictionary::Ptr message = MakeCheckResultMessage(host, cr);
+			listener->SyncSendMessage(sourceEndpoint, message);
+		}

 		return;
 	}
@ -142,21 +226,47 @@ void ClusterEvents::ExecuteCheckFromQueue(const MessageOrigin::Ptr& origin, cons

 	if (command_type == "check_command") {
 		if (!CheckCommand::GetByName(command)) {
-			CheckResult::Ptr cr = new CheckResult();
-			cr->SetState(ServiceUnknown);
-			cr->SetOutput("Check command '" + command + "' does not exist.");
-			Dictionary::Ptr message = MakeCheckResultMessage(host, cr);
-			listener->SyncSendMessage(sourceEndpoint, message);
+			ServiceState state = ServiceUnknown;
+			String output = "Check command '" + command + "' does not exist.";
+			double now = Utility::GetTime();
+			
+			if (params->Contains("source")) {
+				SendEventExecutedCommand(params, state, output, now, now, listener, origin, sourceEndpoint);
+			} else {
+				CheckResult::Ptr cr = new CheckResult();
+				cr->SetState(state);
+				cr->SetOutput(output);
+				Dictionary::Ptr message = MakeCheckResultMessage(host, cr);
+				listener->SyncSendMessage(sourceEndpoint, message);
+			}
+
 			return;
 		}
 	} else if (command_type == "event_command") {
 		if (!EventCommand::GetByName(command)) {
-			Log(LogWarning, "ClusterEvents")
-					<< "Event command '" << command << "' does not exist.";
+			String output = "Event command '" + command + "' does not exist.";
+			Log(LogWarning, "ClusterEvents") << output;
+
+			if (params->Contains("source")) {
+				double now = Utility::GetTime();
+				SendEventExecutedCommand(params, ServiceUnknown, output, now, now, listener, origin, sourceEndpoint);
+			}
+
 			return;
 		}
-	} else
-		return;
+	} else if (command_type == "notification_command") {
+		if (!NotificationCommand::GetByName(command)) {
+			String output = "Notification command '" + command + "' does not exist.";
+			Log(LogWarning, "ClusterEvents") << output;
+
+			if (params->Contains("source")) {
+				double now = Utility::GetTime();
+				SendEventExecutedCommand(params, ServiceUnknown, output, now, now, listener, origin, sourceEndpoint);
+			}
+
+			return;
+		}
+	}

 	attrs->Set(command_type, params->Get("command"));
 	attrs->Set("command_endpoint", sourceEndpoint->GetName());
@ -171,25 +281,74 @@ void ClusterEvents::ExecuteCheckFromQueue(const MessageOrigin::Ptr& origin, cons
 		try {
 			host->ExecuteRemoteCheck(macros);
 		} catch (const std::exception& ex) {
-			CheckResult::Ptr cr = new CheckResult();
-			cr->SetState(ServiceUnknown);
-
 			String output = "Exception occurred while checking '" + host->GetName() + "': " + DiagnosticInformation(ex);
-			cr->SetOutput(output);
-
+			ServiceState state = ServiceUnknown;
 			double now = Utility::GetTime();
-			cr->SetScheduleStart(now);
-			cr->SetScheduleEnd(now);
-			cr->SetExecutionStart(now);
-			cr->SetExecutionEnd(now);

-			Dictionary::Ptr message = MakeCheckResultMessage(host, cr);
-			listener->SyncSendMessage(sourceEndpoint, message);
+			if (params->Contains("source")) {
+				SendEventExecutedCommand(params, state, output, now, now, listener, origin, sourceEndpoint);
+			} else {
+				CheckResult::Ptr cr = new CheckResult();
+				cr->SetState(state);
+				cr->SetOutput(output);
+				cr->SetScheduleStart(now);
+				cr->SetScheduleEnd(now);
+				cr->SetExecutionStart(now);
+				cr->SetExecutionEnd(now);
+
+				Dictionary::Ptr message = MakeCheckResultMessage(host, cr);
+				listener->SyncSendMessage(sourceEndpoint, message);
+			}

 			Log(LogCritical, "checker", output);
 		}
 	} else if (command_type == "event_command") {
-		host->ExecuteEventHandler(macros, true);
+		try {
+			host->ExecuteEventHandler(macros, true);
+		} catch (const std::exception& ex) {
+			if (params->Contains("source")) {
+				String output = "Exception occurred while executing event command '" + command + "' for '" +
+					host->GetName() + "': " + DiagnosticInformation(ex);
+
+				double now = Utility::GetTime();
+				SendEventExecutedCommand(params, ServiceUnknown, output, now, now, listener, origin, sourceEndpoint);
+			} else {
+				throw;
+			}
+		}
+	} else if (command_type == "notification_command" && params->Contains("source")) {
+		/* Get user */
+		User::Ptr user = new User();
+		Dictionary::Ptr attrs = new Dictionary();
+		attrs->Set("__name", params->Get("user"));
+		attrs->Set("type", User::GetTypeName());
+
+		Deserialize(user, attrs, false, FAConfig);
+
+		/* Get notification */
+		Notification::Ptr notification = new Notification();
+		attrs->Clear();
+		attrs->Set("__name", params->Get("notification"));
+		attrs->Set("type", Notification::GetTypeName());
+		attrs->Set("command", command);
+
+		Deserialize(notification, attrs, false, FAConfig);
+
+		try {
+			CheckResult::Ptr cr = new CheckResult();
+			String author = macros->Get("notification_author");
+			NotificationCommand::Ptr notificationCommand = NotificationCommand::GetByName(command);
+
+			notificationCommand->Execute(notification, user, cr, NotificationType::NotificationCustom,
+				author, "");
+		} catch (const std::exception& ex) {
+			String output = "Exception occurred during notification '" + notification->GetName()
+				+ "' for checkable '" + notification->GetCheckable()->GetName()
+				+ "' and user '" + user->GetName() + "' using command '" + command + "': "
+				+ DiagnosticInformation(ex, false);
+			double now = Utility::GetTime();
+			SendEventExecutedCommand(params, ServiceUnknown, output, now, now, listener, origin, sourceEndpoint);
+		}
 	}
 }

--- a/lib/icinga/clusterevents.cpp
+++ b/lib/icinga/clusterevents.cpp
@ -36,6 +36,8 @@ REGISTER_APIFUNCTION(ExecuteCommand, event, &ClusterEvents::ExecuteCommandAPIHan
 REGISTER_APIFUNCTION(SendNotifications, event, &ClusterEvents::SendNotificationsAPIHandler);
 REGISTER_APIFUNCTION(NotificationSentUser, event, &ClusterEvents::NotificationSentUserAPIHandler);
 REGISTER_APIFUNCTION(NotificationSentToAllUsers, event, &ClusterEvents::NotificationSentToAllUsersAPIHandler);
+REGISTER_APIFUNCTION(ExecutedCommand, event, &ClusterEvents::ExecutedCommandAPIHandler);
+REGISTER_APIFUNCTION(UpdateExecutions, event, &ClusterEvents::UpdateExecutionsAPIHandler);

 void ClusterEvents::StaticInitialize()
 {
@ -729,6 +731,66 @@ Value ClusterEvents::AcknowledgementClearedAPIHandler(const MessageOrigin::Ptr&

 Value ClusterEvents::ExecuteCommandAPIHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params)
 {
+	ApiListener::Ptr listener = ApiListener::GetInstance();
+
+	if (!listener)
+		return Empty;
+
+	if (params->Contains("endpoint")) {
+		Endpoint::Ptr execEndpoint = Endpoint::GetByName(params->Get("endpoint"));
+
+		if (execEndpoint != Endpoint::GetLocalEndpoint()) {
+			Zone::Ptr endpointZone = execEndpoint->GetZone();
+			Zone::Ptr localZone = Zone::GetLocalZone();
+
+			if (!endpointZone->IsChildOf(localZone)) {
+				return Empty;
+			}
+
+			/* Check if the child endpoints have Icinga version >= 2.13 */
+			for (const Zone::Ptr &zone : ConfigType::GetObjectsByType<Zone>()) {
+				/* Fetch immediate child zone members */
+				if (zone->GetParent() == localZone && zone->CanAccessObject(endpointZone)) {
+					std::set<Endpoint::Ptr> endpoints = zone->GetEndpoints();
+
+					for (const Endpoint::Ptr &childEndpoint : endpoints) {
+						if (!(childEndpoint->GetCapabilities() & (uint_fast64_t)ApiCapabilities::ExecuteArbitraryCommand)) {
+							double now = Utility::GetTime();
+							Dictionary::Ptr executedParams = new Dictionary();
+							executedParams->Set("execution", params->Get("source"));
+							executedParams->Set("host", params->Get("host"));
+
+							if (params->Contains("service"))
+								executedParams->Set("service", params->Get("service"));
+
+							executedParams->Set("exit", 126);
+							executedParams->Set("output",
+								"Endpoint '" + childEndpoint->GetName() + "' doesn't support executing arbitrary commands.");
+							executedParams->Set("start", now);
+							executedParams->Set("end", now);
+
+							Dictionary::Ptr executedMessage = new Dictionary();
+							executedMessage->Set("jsonrpc", "2.0");
+							executedMessage->Set("method", "event::ExecutedCommand");
+							executedMessage->Set("params", executedParams);
+
+							listener->RelayMessage(nullptr, nullptr, executedMessage, true);
+							return Empty;
+						}
+					}
+				}
+			}
+
+			Dictionary::Ptr execMessage = new Dictionary();
+			execMessage->Set("jsonrpc", "2.0");
+			execMessage->Set("method", "event::ExecuteCommand");
+			execMessage->Set("params", params);
+
+			listener->RelayMessage(origin, endpointZone, execMessage, true);
+			return Empty;
+		}
+	}
+
 	EnqueueCheck(origin, params);

 	return Empty;
@ -1051,3 +1113,172 @@ Value ClusterEvents::NotificationSentToAllUsersAPIHandler(const MessageOrigin::P

 	return Empty;
 }
+
+Value ClusterEvents::ExecutedCommandAPIHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params)
+{
+	ApiListener::Ptr listener = ApiListener::GetInstance();
+
+	if (!listener)
+		return Empty;
+
+	Endpoint::Ptr endpoint;
+
+	if (origin->FromClient) {
+		endpoint = origin->FromClient->GetEndpoint();
+	} else if (origin->IsLocal()) {
+		endpoint = Endpoint::GetLocalEndpoint();
+	}
+
+	if (!endpoint) {
+		Log(LogNotice, "ClusterEvents")
+			<< "Discarding 'update executions API handler' message from '" << origin->FromClient->GetIdentity()
+			<< "': Invalid endpoint origin (client not allowed).";
+
+		return Empty;
+	}
+
+	Host::Ptr host = Host::GetByName(params->Get("host"));
+
+	if (!host)
+		return Empty;
+
+	Checkable::Ptr checkable;
+
+	if (params->Contains("service"))
+		checkable = host->GetServiceByShortName(params->Get("service"));
+	else
+		checkable = host;
+
+	if (!checkable)
+		return Empty;
+
+	ObjectLock oLock (checkable);
+
+	if (origin->FromZone && !origin->FromZone->CanAccessObject(checkable)) {
+		Log(LogNotice, "ClusterEvents")
+			<< "Discarding 'update executions API handler' message for checkable '" << checkable->GetName()
+			<< "' from '" << origin->FromClient->GetIdentity() << "': Unauthorized access.";
+		return Empty;
+	}
+
+	if (!params->Contains("execution")) {
+		Log(LogNotice, "ClusterEvents")
+			<< "Discarding 'update executions API handler' message for checkable '" << checkable->GetName()
+			<< "' from '" << origin->FromClient->GetIdentity() << "': Execution UUID missing.";
+		return Empty;
+	}
+
+	String uuid = params->Get("execution");
+
+	Dictionary::Ptr executions = checkable->GetExecutions();
+
+	if (!executions) {
+		Log(LogNotice, "ClusterEvents")
+			<< "Discarding 'update executions API handler' message for checkable '" << checkable->GetName()
+			<< "' from '" << origin->FromClient->GetIdentity() << "': Execution '" << uuid << "' missing.";
+		return Empty;
+	}
+
+	Dictionary::Ptr execution = executions->Get(uuid);
+
+	if (!execution) {
+		Log(LogNotice, "ClusterEvents")
+			<< "Discarding 'update executions API handler' message for checkable '" << checkable->GetName()
+			<< "' from '" << origin->FromClient->GetIdentity() << "': Execution '" << uuid << "' missing.";
+		return Empty;
+	}
+
+	if (params->Contains("exit"))
+		execution->Set("exit", params->Get("exit"));
+
+	if (params->Contains("output"))
+		execution->Set("output", params->Get("output"));
+
+	if (params->Contains("start"))
+		execution->Set("start", params->Get("start"));
+
+	if (params->Contains("end"))
+		execution->Set("end", params->Get("end"));
+
+	execution->Remove("pending");
+
+	/* Broadcast the update */
+	Dictionary::Ptr executionsToBroadcast = new Dictionary();
+	executionsToBroadcast->Set(uuid, execution);
+	Dictionary::Ptr updateParams = new Dictionary();
+	updateParams->Set("host", host->GetName());
+
+	if (params->Contains("service"))
+		updateParams->Set("service", params->Get("service"));
+
+	updateParams->Set("executions", executionsToBroadcast);
+
+	Dictionary::Ptr updateMessage = new Dictionary();
+	updateMessage->Set("jsonrpc", "2.0");
+	updateMessage->Set("method", "event::UpdateExecutions");
+	updateMessage->Set("params", updateParams);
+
+	listener->RelayMessage(nullptr, checkable, updateMessage, true);
+
+	return Empty;
+}
+
+Value ClusterEvents::UpdateExecutionsAPIHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params)
+{
+	Endpoint::Ptr endpoint = origin->FromClient->GetEndpoint();
+
+	if (!endpoint) {
+		Log(LogNotice, "ClusterEvents")
+			<< "Discarding 'update executions API handler' message from '" << origin->FromClient->GetIdentity()
+			<< "': Invalid endpoint origin (client not allowed).";
+
+		return Empty;
+	}
+
+	Host::Ptr host = Host::GetByName(params->Get("host"));
+
+	if (!host)
+		return Empty;
+
+	Checkable::Ptr checkable;
+
+	if (params->Contains("service"))
+		checkable = host->GetServiceByShortName(params->Get("service"));
+	else
+		checkable = host;
+
+	if (!checkable)
+		return Empty;
+
+	ObjectLock oLock (checkable);
+
+	if (origin->FromZone && !origin->FromZone->CanAccessObject(checkable)) {
+		Log(LogNotice, "ClusterEvents")
+			<< "Discarding 'update executions API handler' message for checkable '" << checkable->GetName()
+			<< "' from '" << origin->FromClient->GetIdentity() << "': Unauthorized access.";
+		return Empty;
+	}
+
+	Dictionary::Ptr executions = checkable->GetExecutions();
+
+	if (!executions)
+		executions = new Dictionary();
+
+	Dictionary::Ptr newExecutions = params->Get("executions");
+	newExecutions->CopyTo(executions);
+	checkable->SetExecutions(executions);
+
+	ApiListener::Ptr listener = ApiListener::GetInstance();
+
+	if (!listener)
+		return Empty;
+
+	Dictionary::Ptr updateMessage = new Dictionary();
+	updateMessage->Set("jsonrpc", "2.0");
+	updateMessage->Set("method", "event::UpdateExecutions");
+	updateMessage->Set("params", params);
+
+	listener->RelayMessage(origin, Zone::GetLocalZone(), updateMessage, true);
+
+	return Empty;
+}
--- a/lib/icinga/clusterevents.hpp
+++ b/lib/icinga/clusterevents.hpp
@ -66,6 +66,8 @@ public:
 	static void NotificationSentToAllUsersHandler(const Notification::Ptr& notification, const Checkable::Ptr& checkable, const std::set<User::Ptr>& users,
 		NotificationType notificationType, const CheckResult::Ptr& cr, const String& author, const String& commentText, const MessageOrigin::Ptr& origin);
 	static Value NotificationSentToAllUsersAPIHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params);
+	static Value ExecutedCommandAPIHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params);
+	static Value UpdateExecutionsAPIHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params);

 	static int GetCheckRequestQueueSize();
 	static void LogRemoteCheckQueueInformation();
--- a/lib/icinga/eventcommand.cpp
+++ b/lib/icinga/eventcommand.cpp
@ -7,6 +7,8 @@ using namespace icinga;

 REGISTER_TYPE(EventCommand);

+thread_local EventCommand::Ptr EventCommand::ExecuteOverride;
+
 void EventCommand::Execute(const Checkable::Ptr& checkable,
 	const Dictionary::Ptr& resolvedMacros, bool useResolvedMacros)
 {
--- a/lib/icinga/eventcommand.hpp
+++ b/lib/icinga/eventcommand.hpp
@ -20,6 +20,8 @@ public:
 	DECLARE_OBJECT(EventCommand);
 	DECLARE_OBJECTNAME(EventCommand);

+	static thread_local EventCommand::Ptr ExecuteOverride;
+
 	virtual void Execute(const Checkable::Ptr& checkable,
 		const Dictionary::Ptr& resolvedMacros = nullptr,
 		bool useResolvedMacros = false);
--- a/lib/icinga/host.ti
+++ b/lib/icinga/host.ti
@ -21,10 +21,10 @@ class Host : Checkable

 	[config] String display_name {
 		get {{{
-			if (m_DisplayName.IsEmpty())
+			if (m_DisplayName.m_Value.IsEmpty())
 				return GetName();
 			else
-				return m_DisplayName;
+				return m_DisplayName.m_Value;
 		}}}
 	};

--- a/lib/icinga/hostgroup.ti
+++ b/lib/icinga/hostgroup.ti
@ -11,10 +11,10 @@ class HostGroup : CustomVarObject
 {
 	[config] String display_name {
 		get {{{
-			if (m_DisplayName.IsEmpty())
+			if (m_DisplayName.m_Value.IsEmpty())
 				return GetName();
 			else
-				return m_DisplayName;
+				return m_DisplayName.m_Value;
 		}}}
 	};

--- a/lib/icinga/macroprocessor.cpp
+++ b/lib/icinga/macroprocessor.cpp
@ -15,6 +15,8 @@

 using namespace icinga;

+thread_local Dictionary::Ptr MacroResolver::OverrideMacros;
+
 Value MacroProcessor::ResolveMacros(const Value& str, const ResolverList& resolvers,
 	const CheckResult::Ptr& cr, String *missingMacro,
 	const MacroProcessor::EscapeCallback& escapeFn, const Dictionary::Ptr& resolvedMacros,
--- a/lib/icinga/macroresolver.hpp
+++ b/lib/icinga/macroresolver.hpp
@ -21,6 +21,8 @@ class MacroResolver
 public:
 	DECLARE_PTR_TYPEDEFS(MacroResolver);

+	static thread_local Dictionary::Ptr OverrideMacros;
+
 	virtual bool ResolveMacro(const String& macro, const CheckResult::Ptr& cr, Value *result) const = 0;
 };

--- a/lib/icinga/notificationcommand.cpp
+++ b/lib/icinga/notificationcommand.cpp
@ -7,6 +7,8 @@ using namespace icinga;

 REGISTER_TYPE(NotificationCommand);

+thread_local NotificationCommand::Ptr NotificationCommand::ExecuteOverride;
+
 Dictionary::Ptr NotificationCommand::Execute(const Notification::Ptr& notification,
 	const User::Ptr& user, const CheckResult::Ptr& cr, const NotificationType& type,
 	const String& author, const String& comment, const Dictionary::Ptr& resolvedMacros,
--- a/lib/icinga/notificationcommand.hpp
+++ b/lib/icinga/notificationcommand.hpp
@ -22,6 +22,8 @@ public:
 	DECLARE_OBJECT(NotificationCommand);
 	DECLARE_OBJECTNAME(NotificationCommand);

+	static thread_local NotificationCommand::Ptr ExecuteOverride;
+
 	virtual Dictionary::Ptr Execute(const intrusive_ptr<Notification>& notification,
 		const User::Ptr& user, const CheckResult::Ptr& cr, const NotificationType& type,
 		const String& author, const String& comment,
--- a/lib/icinga/service.ti
+++ b/lib/icinga/service.ti
@ -33,10 +33,10 @@ class Service : Checkable < ServiceNameComposer

 	[config] String display_name {
 		get {{{
-			if (m_DisplayName.IsEmpty())
+			if (m_DisplayName.m_Value.IsEmpty())
 				return GetShortName();
 			else
-				return m_DisplayName;
+				return m_DisplayName.m_Value;
 		}}}
 	};
 	[config, required] name(Host) host_name;
--- a/lib/icinga/servicegroup.ti
+++ b/lib/icinga/servicegroup.ti
@ -11,10 +11,10 @@ class ServiceGroup : CustomVarObject
 {
 	[config] String display_name {
 		get {{{
-			if (m_DisplayName.IsEmpty())
+			if (m_DisplayName.m_Value.IsEmpty())
 				return GetName();
 			else
-				return m_DisplayName;
+				return m_DisplayName.m_Value;
 		}}}
 	};

--- a/lib/icinga/timeperiod.ti
+++ b/lib/icinga/timeperiod.ti
@ -12,10 +12,10 @@ class TimePeriod : CustomVarObject
 {
 	[config] String display_name {
 		get {{{
-			if (m_DisplayName.IsEmpty())
+			if (m_DisplayName.m_Value.IsEmpty())
 				return GetName();
 			else
-				return m_DisplayName;
+				return m_DisplayName.m_Value;
 		}}}
 	};
 	[config] Dictionary::Ptr ranges;
--- a/lib/icinga/user.ti
+++ b/lib/icinga/user.ti
@ -13,10 +13,10 @@ class User : CustomVarObject
 {
 	[config] String display_name {
 		get {{{
-			if (m_DisplayName.IsEmpty())
+			if (m_DisplayName.m_Value.IsEmpty())
 				return GetName();
 			else
-				return m_DisplayName;
+				return m_DisplayName.m_Value;
 		}}}
 	};
 	[config, no_user_modify, required] array(name(UserGroup)) groups {
--- a/lib/icinga/usergroup.ti
+++ b/lib/icinga/usergroup.ti
@ -11,10 +11,10 @@ class UserGroup : CustomVarObject
 {
 	[config] String display_name {
 		get {{{
-			if (m_DisplayName.IsEmpty())
+			if (m_DisplayName.m_Value.IsEmpty())
 				return GetName();
 			else
-				return m_DisplayName;
+				return m_DisplayName.m_Value;
 		}}}
 	};

--- a/lib/methods/clusterchecktask.cpp
+++ b/lib/methods/clusterchecktask.cpp
@ -28,46 +28,72 @@ void ClusterCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRe
 	if (resolvedMacros && !useResolvedMacros)
 		return;

-	CheckCommand::Ptr command = checkable->GetCheckCommand();
-	cr->SetCommand(command->GetName());
+	CheckCommand::Ptr command = CheckCommand::ExecuteOverride ? CheckCommand::ExecuteOverride : checkable->GetCheckCommand();
+	String commandName = command->GetName();

 	ApiListener::Ptr listener = ApiListener::GetInstance();
-
 	if (!listener) {
-		cr->SetOutput("No API listener is configured for this instance.");
-		cr->SetState(ServiceUnknown);
-		checkable->ProcessCheckResult(cr);
+		String output = "No API listener is configured for this instance.";
+
+		if (Checkable::ExecuteCommandProcessFinishedHandler) {
+			double now = Utility::GetTime();
+			ProcessResult pr;
+			pr.PID = -1;
+			pr.ExecutionStart = now;
+			pr.ExecutionEnd = now;
+			pr.ExitStatus = 126;
+			pr.Output = output;
+			Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+		} else {
+			cr->SetOutput(output);
+			cr->SetState(ServiceUnknown);
+			checkable->ProcessCheckResult(cr);
+		}
+
 		return;
 	}

 	std::pair<Dictionary::Ptr, Dictionary::Ptr> stats = listener->GetStatus();
-
 	Dictionary::Ptr status = stats.first;
-
-	/* use feature stats perfdata */
-	std::pair<Dictionary::Ptr, Array::Ptr> feature_stats = CIB::GetFeatureStats();
-	cr->SetPerformanceData(feature_stats.second);
-
 	int numConnEndpoints = status->Get("num_conn_endpoints");
 	int numNotConnEndpoints = status->Get("num_not_conn_endpoints");

+	ServiceState state;
 	String output = "Icinga 2 Cluster";

 	if (numNotConnEndpoints > 0) {
 		output += " Problem: " + Convert::ToString(numNotConnEndpoints) + " endpoints are not connected.";
 		output += "\n(" + FormatArray(status->Get("not_conn_endpoints")) + ")";

-		cr->SetState(ServiceCritical);
+		state = ServiceCritical;
 	} else {
 		output += " OK: " + Convert::ToString(numConnEndpoints) + " endpoints are connected.";
 		output += "\n(" + FormatArray(status->Get("conn_endpoints")) + ")";

-		cr->SetState(ServiceOK);
+		state = ServiceOK;
 	}

-	cr->SetOutput(output);
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		double now = Utility::GetTime();
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = output;
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = state;

-	checkable->ProcessCheckResult(cr);
+		Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+	} else {
+		/* use feature stats perfdata */
+		std::pair<Dictionary::Ptr, Array::Ptr> feature_stats = CIB::GetFeatureStats();
+		cr->SetPerformanceData(feature_stats.second);
+
+		cr->SetCommand(commandName);
+		cr->SetState(state);
+		cr->SetOutput(output);
+
+		checkable->ProcessCheckResult(cr);
+	}
 }

 String ClusterCheckTask::FormatArray(const Array::Ptr& arr)
--- a/lib/methods/clusterzonechecktask.cpp
+++ b/lib/methods/clusterzonechecktask.cpp
@ -21,15 +21,33 @@ void ClusterZoneCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const Che
 	REQUIRE_NOT_NULL(cr);

 	ApiListener::Ptr listener = ApiListener::GetInstance();
+	CheckCommand::Ptr command = CheckCommand::ExecuteOverride ? CheckCommand::ExecuteOverride : checkable->GetCheckCommand();
+	String commandName = command->GetName();

 	if (!listener) {
-		cr->SetOutput("No API listener is configured for this instance.");
-		cr->SetState(ServiceUnknown);
-		checkable->ProcessCheckResult(cr);
+		String output = "No API listener is configured for this instance.";
+		ServiceState state = ServiceUnknown;
+
+		if (Checkable::ExecuteCommandProcessFinishedHandler) {
+			double now = Utility::GetTime();
+			ProcessResult pr;
+			pr.PID = -1;
+			pr.Output = output;
+			pr.ExecutionStart = now;
+			pr.ExecutionEnd = now;
+			pr.ExitStatus = state;
+
+			Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+		} else {
+			cr->SetCommand(commandName);
+			cr->SetOutput(output);
+			cr->SetState(state);
+			checkable->ProcessCheckResult(cr);
+		}
+
 		return;
 	}

-	CheckCommand::Ptr command = checkable->GetCheckCommand();
 	Value raw_command = command->GetCommandLine();

 	Host::Ptr host;
@ -37,6 +55,10 @@ void ClusterZoneCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const Che
 	tie(host, service) = GetHostService(checkable);

 	MacroProcessor::ResolverList resolvers;
+
+	if (MacroResolver::OverrideMacros)
+		resolvers.emplace_back("override", MacroResolver::OverrideMacros);
+
 	if (service)
 		resolvers.emplace_back("service", service);
 	resolvers.emplace_back("host", host);
@ -58,21 +80,52 @@ void ClusterZoneCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const Che
 	if (resolvedMacros && !useResolvedMacros)
 		return;

-	cr->SetCommand(command->GetName());
-
 	if (zoneName.IsEmpty()) {
-		cr->SetOutput("Macro 'cluster_zone' must be set.");
-		cr->SetState(ServiceUnknown);
-		checkable->ProcessCheckResult(cr);
+		String output = "Macro 'cluster_zone' must be set.";
+		ServiceState state = ServiceUnknown;
+
+		if (Checkable::ExecuteCommandProcessFinishedHandler) {
+			double now = Utility::GetTime();
+			ProcessResult pr;
+			pr.PID = -1;
+			pr.Output = output;
+			pr.ExecutionStart = now;
+			pr.ExecutionEnd = now;
+			pr.ExitStatus = state;
+
+			Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+		} else {
+			cr->SetCommand(commandName);
+			cr->SetOutput(output);
+			cr->SetState(state);
+			checkable->ProcessCheckResult(cr);
+		}
+
 		return;
 	}

 	Zone::Ptr zone = Zone::GetByName(zoneName);

 	if (!zone) {
-		cr->SetOutput("Zone '" + zoneName + "' does not exist.");
-		cr->SetState(ServiceUnknown);
-		checkable->ProcessCheckResult(cr);
+		String output = "Zone '" + zoneName + "' does not exist.";
+		ServiceState state = ServiceUnknown;
+
+		if (Checkable::ExecuteCommandProcessFinishedHandler) {
+			double now = Utility::GetTime();
+			ProcessResult pr;
+			pr.PID = -1;
+			pr.Output = output;
+			pr.ExecutionStart = now;
+			pr.ExecutionEnd = now;
+			pr.ExitStatus = state;
+
+			Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+		} else {
+			cr->SetCommand(commandName);
+			cr->SetOutput(output);
+			cr->SetState(state);
+			checkable->ProcessCheckResult(cr);
+		}
 		return;
 	}

@ -107,34 +160,52 @@ void ClusterZoneCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const Che
 		bytesReceivedPerSecond += endpoint->GetBytesReceivedPerSecond();
 	}

+	ServiceState state;
+	String output;
+
 	if (connected) {
-		cr->SetState(ServiceOK);
-		cr->SetOutput("Zone '" + zoneName + "' is connected. Log lag: " + Utility::FormatDuration(zoneLag));
+		state = ServiceOK;
+		output = "Zone '" + zoneName + "' is connected. Log lag: " + Utility::FormatDuration(zoneLag);

 		/* Check whether the thresholds have been resolved and compare them */
 		if (missingLagCritical.IsEmpty() && zoneLag > lagCritical) {
-			cr->SetState(ServiceCritical);
-			cr->SetOutput("Zone '" + zoneName + "' is connected. Log lag: " + Utility::FormatDuration(zoneLag)
-				+ " greater than critical threshold: " + Utility::FormatDuration(lagCritical));
+			state = ServiceCritical;
+			output = "Zone '" + zoneName + "' is connected. Log lag: " + Utility::FormatDuration(zoneLag)
+				+ " greater than critical threshold: " + Utility::FormatDuration(lagCritical);
 		} else if (missingLagWarning.IsEmpty() && zoneLag > lagWarning) {
-			cr->SetState(ServiceWarning);
-			cr->SetOutput("Zone '" + zoneName + "' is connected. Log lag: " + Utility::FormatDuration(zoneLag)
-				+ " greater than warning threshold: " + Utility::FormatDuration(lagWarning));
+			state = ServiceWarning;
+			output = "Zone '" + zoneName + "' is connected. Log lag: " + Utility::FormatDuration(zoneLag)
+				+ " greater than warning threshold: " + Utility::FormatDuration(lagWarning);
 		}
 	} else {
-		cr->SetState(ServiceCritical);
-		cr->SetOutput("Zone '" + zoneName + "' is not connected. Log lag: " + Utility::FormatDuration(zoneLag));
+		state = ServiceCritical;
+		output = "Zone '" + zoneName + "' is not connected. Log lag: " + Utility::FormatDuration(zoneLag);
 	}

-	cr->SetPerformanceData(new Array({
-		new PerfdataValue("slave_lag", zoneLag, false, "s", lagWarning, lagCritical),
-		new PerfdataValue("last_messages_sent", lastMessageSent),
-		new PerfdataValue("last_messages_received", lastMessageReceived),
-		new PerfdataValue("sum_messages_sent_per_second", messagesSentPerSecond),
-		new PerfdataValue("sum_messages_received_per_second", messagesReceivedPerSecond),
-		new PerfdataValue("sum_bytes_sent_per_second", bytesSentPerSecond),
-		new PerfdataValue("sum_bytes_received_per_second", bytesReceivedPerSecond)
-	}));
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		double now = Utility::GetTime();
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = output;
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = state;

-	checkable->ProcessCheckResult(cr);
+		Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+	} else {
+		cr->SetCommand(commandName);
+		cr->SetState(state);
+		cr->SetOutput(output);
+		cr->SetPerformanceData(new Array({
+			new PerfdataValue("slave_lag", zoneLag, false, "s", lagWarning, lagCritical),
+			new PerfdataValue("last_messages_sent", lastMessageSent),
+			new PerfdataValue("last_messages_received", lastMessageReceived),
+			new PerfdataValue("sum_messages_sent_per_second", messagesSentPerSecond),
+			new PerfdataValue("sum_messages_received_per_second", messagesReceivedPerSecond),
+			new PerfdataValue("sum_bytes_sent_per_second", bytesSentPerSecond),
+			new PerfdataValue("sum_bytes_received_per_second", bytesReceivedPerSecond)
+		}));
+
+		checkable->ProcessCheckResult(cr);
+	}
 }
--- a/lib/methods/dummychecktask.cpp
+++ b/lib/methods/dummychecktask.cpp
@ -22,13 +22,17 @@ void DummyCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResu
 	REQUIRE_NOT_NULL(checkable);
 	REQUIRE_NOT_NULL(cr);

-	CheckCommand::Ptr command = checkable->GetCheckCommand();
+	CheckCommand::Ptr command = CheckCommand::ExecuteOverride ? CheckCommand::ExecuteOverride : checkable->GetCheckCommand();

 	Host::Ptr host;
 	Service::Ptr service;
 	tie(host, service) = GetHostService(checkable);

 	MacroProcessor::ResolverList resolvers;
+
+	if (MacroResolver::OverrideMacros)
+		resolvers.emplace_back("override", MacroResolver::OverrideMacros);
+
 	if (service)
 		resolvers.emplace_back("service", service);
 	resolvers.emplace_back("host", host);
@ -48,14 +52,26 @@ void DummyCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResu
 	std::pair<String, String> co = PluginUtility::ParseCheckOutput(dummyText);

 	double now = Utility::GetTime();
+	String commandName = command->GetName();

-	cr->SetOutput(co.first);
-	cr->SetPerformanceData(PluginUtility::SplitPerfdata(co.second));
-	cr->SetState(PluginUtility::ExitStatusToState(dummyState));
-	cr->SetExitStatus(dummyState);
-	cr->SetExecutionStart(now);
-	cr->SetExecutionEnd(now);
-	cr->SetCommand(command->GetName());
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = dummyText;
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = dummyState;

-	checkable->ProcessCheckResult(cr);
+		Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+	} else {
+		cr->SetOutput(co.first);
+		cr->SetPerformanceData(PluginUtility::SplitPerfdata(co.second));
+		cr->SetState(PluginUtility::ExitStatusToState(dummyState));
+		cr->SetExitStatus(dummyState);
+		cr->SetExecutionStart(now);
+		cr->SetExecutionEnd(now);
+		cr->SetCommand(commandName);
+
+		checkable->ProcessCheckResult(cr);
+	}
 }
--- a/lib/methods/exceptionchecktask.cpp
+++ b/lib/methods/exceptionchecktask.cpp
@ -23,5 +23,19 @@ void ExceptionCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const Check
 	if (resolvedMacros && !useResolvedMacros)
 		return;

-	BOOST_THROW_EXCEPTION(ScriptError("Test") << boost::errinfo_api_function("Test"));
+	ScriptError scriptError = ScriptError("Test") << boost::errinfo_api_function("Test");
+
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		double now = Utility::GetTime();
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = scriptError.what();
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = 3;
+
+		Checkable::ExecuteCommandProcessFinishedHandler("", pr);
+	} else {
+		BOOST_THROW_EXCEPTION(ScriptError("Test") << boost::errinfo_api_function("Test"));
+	}
 }
--- a/lib/methods/icingachecktask.cpp
+++ b/lib/methods/icingachecktask.cpp
@ -26,13 +26,17 @@ void IcingaCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
 	REQUIRE_NOT_NULL(checkable);
 	REQUIRE_NOT_NULL(cr);

-	CheckCommand::Ptr command = checkable->GetCheckCommand();
+	CheckCommand::Ptr command = CheckCommand::ExecuteOverride ? CheckCommand::ExecuteOverride : checkable->GetCheckCommand();

 	Host::Ptr host;
 	Service::Ptr service;
 	tie(host, service) = GetHostService(checkable);

 	MacroProcessor::ResolverList resolvers;
+
+	if (MacroResolver::OverrideMacros)
+		resolvers.emplace_back("override", MacroResolver::OverrideMacros);
+
 	if (service)
 		resolvers.emplace_back("service", service);
 	resolvers.emplace_back("host", host);
@ -148,7 +152,7 @@ void IcingaCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
 	perfdata->Add(new PerfdataValue("sum_bytes_received_per_second", bytesReceivedPerSecond));

 	cr->SetPerformanceData(perfdata);
-	cr->SetState(ServiceOK);
+	ServiceState state = ServiceOK;

 	String appVersion = Application::GetAppVersion();

@ -160,7 +164,7 @@ void IcingaCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes

 	if (lastReloadFailed > 0) {
 		output += "; Last reload attempt failed at " + Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", lastReloadFailed);
-		cr->SetState(ServiceWarning);
+		state =ServiceWarning;
 	}

 	/* Indicate a warning when the last synced config caused a stage validation error. */
@ -173,7 +177,7 @@ void IcingaCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
 			output += "; Last zone sync stage validation failed at "
 				+ Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", validationResult->Get("ts"));

-			cr->SetState(ServiceWarning);
+			state = ServiceWarning;
 		}
 	}

@ -182,11 +186,26 @@ void IcingaCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
 	/* Return an error if the version is less than specified (optional). */
 	if (missingIcingaMinVersion.IsEmpty() && !icingaMinVersion.IsEmpty() && Utility::CompareVersion(icingaMinVersion, parsedAppVersion) < 0) {
 		output += "; Minimum version " + icingaMinVersion + " is not installed.";
-		cr->SetState(ServiceCritical);
+		state = ServiceCritical;
 	}

-	cr->SetOutput(output);
-	cr->SetCommand(command->GetName());
+	String commandName = command->GetName();

-	checkable->ProcessCheckResult(cr);
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		double now = Utility::GetTime();
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = output;
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = state;
+
+		Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+	} else {
+		cr->SetState(state);
+		cr->SetOutput(output);
+		cr->SetCommand(commandName);
+
+		checkable->ProcessCheckResult(cr);
+	}
 }
--- a/lib/methods/nullchecktask.cpp
+++ b/lib/methods/nullchecktask.cpp
@ -26,12 +26,25 @@ void NullCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResul

 	String output = "Hello from ";
 	output += IcingaApplication::GetInstance()->GetNodeName();
+	ServiceState state = ServiceOK;

-	cr->SetOutput(output);
-	cr->SetPerformanceData(new Array({
-		new PerfdataValue("time", Convert::ToDouble(Utility::GetTime()))
-	}));
-	cr->SetState(ServiceOK);
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		double now = Utility::GetTime();
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = output;
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = state;

-	checkable->ProcessCheckResult(cr);
+		Checkable::ExecuteCommandProcessFinishedHandler("", pr);
+	} else {
+		cr->SetOutput(output);
+		cr->SetPerformanceData(new Array({
+			new PerfdataValue("time", Convert::ToDouble(Utility::GetTime()))
+		}));
+		cr->SetState(state);
+
+		checkable->ProcessCheckResult(cr);
+	}
 }
--- a/lib/methods/nulleventtask.cpp
+++ b/lib/methods/nulleventtask.cpp
@ -11,4 +11,16 @@ REGISTER_FUNCTION_NONCONST(Internal, NullEvent, &NullEventTask::ScriptFunc, "che
 void NullEventTask::ScriptFunc(const Checkable::Ptr& checkable, const Dictionary::Ptr& resolvedMacros, bool useResolvedMacros)
 {
 	REQUIRE_NOT_NULL(checkable);
+
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		double now = Utility::GetTime();
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = "";
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = 0;
+
+		Checkable::ExecuteCommandProcessFinishedHandler("", pr);
+	}
 }
--- a/lib/methods/pluginchecktask.cpp
+++ b/lib/methods/pluginchecktask.cpp
@ -22,13 +22,17 @@ void PluginCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
 	REQUIRE_NOT_NULL(checkable);
 	REQUIRE_NOT_NULL(cr);

-	CheckCommand::Ptr commandObj = checkable->GetCheckCommand();
+	CheckCommand::Ptr commandObj = CheckCommand::ExecuteOverride ? CheckCommand::ExecuteOverride : checkable->GetCheckCommand();

 	Host::Ptr host;
 	Service::Ptr service;
 	tie(host, service) = GetHostService(checkable);

 	MacroProcessor::ResolverList resolvers;
+
+	if (MacroResolver::OverrideMacros)
+		resolvers.emplace_back("override", MacroResolver::OverrideMacros);
+
 	if (service)
 		resolvers.emplace_back("service", service);
 	resolvers.emplace_back("host", host);
@ -40,9 +44,16 @@ void PluginCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
 	if (!checkable->GetCheckTimeout().IsEmpty())
 		timeout = checkable->GetCheckTimeout();

+	std::function<void(const Value& commandLine, const ProcessResult&)> callback;
+
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		callback = Checkable::ExecuteCommandProcessFinishedHandler;
+	} else {
+		callback = std::bind(&PluginCheckTask::ProcessFinishedHandler, checkable, cr, _1, _2);
+	}
+
 	PluginUtility::ExecuteCommand(commandObj, checkable, checkable->GetLastCheckResult(),
-		resolvers, resolvedMacros, useResolvedMacros, timeout,
-		std::bind(&PluginCheckTask::ProcessFinishedHandler, checkable, cr, _1, _2));
+		resolvers, resolvedMacros, useResolvedMacros, timeout, callback);

 	if (!resolvedMacros || useResolvedMacros) {
 		Checkable::CurrentConcurrentChecks.fetch_add(1);
--- a/lib/methods/plugineventtask.cpp
+++ b/lib/methods/plugineventtask.cpp
@ -21,13 +21,17 @@ void PluginEventTask::ScriptFunc(const Checkable::Ptr& checkable,
 {
 	REQUIRE_NOT_NULL(checkable);

-	EventCommand::Ptr commandObj = checkable->GetEventCommand();
+	EventCommand::Ptr commandObj = EventCommand::ExecuteOverride ? EventCommand::ExecuteOverride : checkable->GetEventCommand();

 	Host::Ptr host;
 	Service::Ptr service;
 	tie(host, service) = GetHostService(checkable);

 	MacroProcessor::ResolverList resolvers;
+
+	if (MacroResolver::OverrideMacros)
+		resolvers.emplace_back("override", MacroResolver::OverrideMacros);
+
 	if (service)
 		resolvers.emplace_back("service", service);
 	resolvers.emplace_back("host", host);
@ -35,10 +39,16 @@ void PluginEventTask::ScriptFunc(const Checkable::Ptr& checkable,
 	resolvers.emplace_back("icinga", IcingaApplication::GetInstance());

 	int timeout = commandObj->GetTimeout();
+	std::function<void(const Value& commandLine, const ProcessResult&)> callback;
+
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		callback = Checkable::ExecuteCommandProcessFinishedHandler;
+	} else {
+		callback = std::bind(&PluginEventTask::ProcessFinishedHandler, checkable, _1, _2);
+	}

 	PluginUtility::ExecuteCommand(commandObj, checkable, checkable->GetLastCheckResult(),
-		resolvers, resolvedMacros, useResolvedMacros, timeout,
-		std::bind(&PluginEventTask::ProcessFinishedHandler, checkable, _1, _2));
+		resolvers, resolvedMacros, useResolvedMacros, timeout, callback);
 }

 void PluginEventTask::ProcessFinishedHandler(const Checkable::Ptr& checkable, const Value& commandLine, const ProcessResult& pr)
--- a/lib/methods/pluginnotificationtask.cpp
+++ b/lib/methods/pluginnotificationtask.cpp
@ -25,7 +25,7 @@ void PluginNotificationTask::ScriptFunc(const Notification::Ptr& notification,
 	REQUIRE_NOT_NULL(notification);
 	REQUIRE_NOT_NULL(user);

-	NotificationCommand::Ptr commandObj = notification->GetCommand();
+	NotificationCommand::Ptr commandObj = NotificationCommand::ExecuteOverride ? NotificationCommand::ExecuteOverride : notification->GetCommand();

 	auto type = static_cast<NotificationType>(itype);

@ -42,6 +42,10 @@ void PluginNotificationTask::ScriptFunc(const Notification::Ptr& notification,
 	tie(host, service) = GetHostService(checkable);

 	MacroProcessor::ResolverList resolvers;
+
+	if (MacroResolver::OverrideMacros)
+		resolvers.emplace_back("override", MacroResolver::OverrideMacros);
+
 	resolvers.emplace_back("user", user);
 	resolvers.emplace_back("notification", notificationExtra);
 	resolvers.emplace_back("notification", notification);
@ -52,10 +56,16 @@ void PluginNotificationTask::ScriptFunc(const Notification::Ptr& notification,
 	resolvers.emplace_back("icinga", IcingaApplication::GetInstance());

 	int timeout = commandObj->GetTimeout();
+	std::function<void(const Value& commandLine, const ProcessResult&)> callback;
+
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		callback = Checkable::ExecuteCommandProcessFinishedHandler;
+	} else {
+		callback = std::bind(&PluginNotificationTask::ProcessFinishedHandler, checkable, _1, _2);
+	}

 	PluginUtility::ExecuteCommand(commandObj, checkable, cr, resolvers,
-		resolvedMacros, useResolvedMacros, timeout,
-		std::bind(&PluginNotificationTask::ProcessFinishedHandler, checkable, _1, _2));
+		resolvedMacros, useResolvedMacros, timeout, callback);
 }

 void PluginNotificationTask::ProcessFinishedHandler(const Checkable::Ptr& checkable, const Value& commandLine, const ProcessResult& pr)
--- a/lib/methods/randomchecktask.cpp
+++ b/lib/methods/randomchecktask.cpp
@ -31,22 +31,35 @@ void RandomCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
 		+ ". Icinga 2 has been running for " + Utility::FormatDuration(uptime)
 		+ ". Version: " + Application::GetAppVersion();

-	cr->SetOutput(output);
+	CheckCommand::Ptr command = CheckCommand::ExecuteOverride ? CheckCommand::ExecuteOverride : checkable->GetCheckCommand();
+	String commandName = command->GetName();
+	ServiceState state = static_cast<ServiceState>(Utility::Random() % 4);

-	double random = Utility::Random() % 1000;
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		double now = Utility::GetTime();
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = output;
+		pr.ExecutionStart = now;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = state;

-	cr->SetPerformanceData(new Array({
-		new PerfdataValue("time", now),
-		new PerfdataValue("value", random),
-		new PerfdataValue("value_1m", random * 0.9),
-		new PerfdataValue("value_5m", random * 0.8),
-		new PerfdataValue("uptime", uptime),
-	}));
+		Checkable::ExecuteCommandProcessFinishedHandler(commandName, pr);
+	} else {
+		cr->SetOutput(output);

-	cr->SetState(static_cast<ServiceState>(Utility::Random() % 4));
+		double random = Utility::Random() % 1000;
+		cr->SetPerformanceData(new Array({
+			new PerfdataValue("time", now),
+			new PerfdataValue("value", random),
+			new PerfdataValue("value_1m", random * 0.9),
+			new PerfdataValue("value_5m", random * 0.8),
+			new PerfdataValue("uptime", uptime),
+		}));

-	CheckCommand::Ptr command = checkable->GetCheckCommand();
-	cr->SetCommand(command->GetName());
+		cr->SetState(state);
+		cr->SetCommand(commandName);

-	checkable->ProcessCheckResult(cr);
+		checkable->ProcessCheckResult(cr);
+	}
 }
--- a/lib/methods/sleepchecktask.cpp
+++ b/lib/methods/sleepchecktask.cpp
@ -18,13 +18,17 @@ void SleepCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResu
    REQUIRE_NOT_NULL(checkable);
    REQUIRE_NOT_NULL(cr);

-    CheckCommand::Ptr commandObj = checkable->GetCheckCommand();
+	CheckCommand::Ptr commandObj = CheckCommand::ExecuteOverride ? CheckCommand::ExecuteOverride : checkable->GetCheckCommand();

    Host::Ptr host;
    Service::Ptr service;
    tie(host, service) = GetHostService(checkable);

-    MacroProcessor::ResolverList resolvers;
+	MacroProcessor::ResolverList resolvers;
+
+	if (MacroResolver::OverrideMacros)
+		resolvers.emplace_back("override", MacroResolver::OverrideMacros);
+
    if (service)
        resolvers.emplace_back("service", service);
    resolvers.emplace_back("host", host);
@ -42,13 +46,24 @@ void SleepCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckResu
    String output = "Slept for " + Convert::ToString(sleepTime) + " seconds.";

    double now = Utility::GetTime();
+	CheckCommand::Ptr command = checkable->GetCheckCommand();
+	String commandName = command->GetName();

-    cr->SetOutput(output);
-    cr->SetExecutionStart(now);
-    cr->SetExecutionEnd(now);
+	if (Checkable::ExecuteCommandProcessFinishedHandler) {
+		ProcessResult pr;
+		pr.PID = -1;
+		pr.Output = output;
+		pr.ExecutionStart = now - sleepTime;
+		pr.ExecutionEnd = now;
+		pr.ExitStatus = 0;

-    CheckCommand::Ptr command = checkable->GetCheckCommand();
-    cr->SetCommand(command->GetName());
+		Checkable::ExecuteCommandProcessFinishedHandler("", pr);
+	} else {
+		cr->SetOutput(output);
+		cr->SetExecutionStart(now);
+		cr->SetExecutionEnd(now);
+		cr->SetCommand(commandName);

-    checkable->ProcessCheckResult(cr);
+		checkable->ProcessCheckResult(cr);
+	}
 }
--- a/lib/remote/actionshandler.cpp
+++ b/lib/remote/actionshandler.cpp
@ -4,12 +4,15 @@
 #include "remote/httputility.hpp"
 #include "remote/filterutility.hpp"
 #include "remote/apiaction.hpp"
+#include "base/defer.hpp"
 #include "base/exception.hpp"
 #include "base/logger.hpp"
 #include <set>

 using namespace icinga;

+thread_local ApiUser::Ptr ActionsHandler::AuthenticatedApiUser;
+
 REGISTER_URLHANDLER("/v1/actions", ActionsHandler);

 bool ActionsHandler::HandleRequest(
@ -71,6 +74,11 @@ bool ActionsHandler::HandleRequest(

 	bool verbose = false;

+	ActionsHandler::AuthenticatedApiUser = user;
+	Defer a ([]() {
+		ActionsHandler::AuthenticatedApiUser = nullptr;
+	});
+
 	if (params)
 		verbose = HttpUtility::GetLastParameter(params, "verbose");

@ -94,7 +102,7 @@ bool ActionsHandler::HandleRequest(
 	int statusCode = 500;

 	for (const Dictionary::Ptr& res : results) {
-		if (res->Contains("code") && res->Get("code") == 200) {
+		if (res->Contains("code") && res->Get("code") >= 200 && res->Get("code") <= 299) {
 			statusCode = 200;
 			break;
 		}
--- a/lib/remote/actionshandler.hpp
+++ b/lib/remote/actionshandler.hpp
@ -13,6 +13,8 @@ class ActionsHandler final : public HttpHandler
 public:
 	DECLARE_PTR_TYPEDEFS(ActionsHandler);

+	static thread_local ApiUser::Ptr AuthenticatedApiUser;
+
 	bool HandleRequest(
 		AsioTlsStream& stream,
 		const ApiUser::Ptr& user,
--- a/lib/remote/apilistener.cpp
+++ b/lib/remote/apilistener.cpp
@ -29,6 +29,8 @@
 #include <boost/asio/spawn.hpp>
 #include <boost/asio/ssl/context.hpp>
 #include <boost/date_time/posix_time/posix_time_duration.hpp>
+#include <boost/lexical_cast.hpp>
+#include <boost/regex.hpp>
 #include <boost/system/error_code.hpp>
 #include <climits>
 #include <cstdint>
@ -506,6 +508,22 @@ void ApiListener::NewClientHandler(
 	}
 }

+static const auto l_AppVersionInt (([]() -> unsigned long {
+	auto appVersion (Application::GetAppVersion());
+	boost::regex rgx (R"EOF(^v?(\d+)\.(\d+)\.(\d+))EOF");
+	boost::smatch match;
+
+	if (!boost::regex_search(appVersion.GetData(), match, rgx)) {
+		return 0;
+	}
+
+	return 100u * 100u * boost::lexical_cast<unsigned long>(match[1].str())
+		+ 100u * boost::lexical_cast<unsigned long>(match[2].str())
+		+ boost::lexical_cast<unsigned long>(match[3].str());
+})());
+
+static const auto l_MyCapabilities (ApiCapabilities::ExecuteArbitraryCommand);
+
 /**
 * Processes a new client connection.
 *
@ -637,7 +655,10 @@ void ApiListener::NewClientHandlerInternal(
 		JsonRpc::SendMessage(client, new Dictionary({
 			{ "jsonrpc", "2.0" },
 			{ "method", "icinga::Hello" },
-			{ "params", new Dictionary() }
+			{ "params", new Dictionary({
+				{ "version", (double)l_AppVersionInt },
+				{ "capabilities", (double)l_MyCapabilities }
+			}) }
 		}), yc);

 		client->async_flush(yc);
@ -670,6 +691,17 @@ void ApiListener::NewClientHandlerInternal(
 		}

 		if (firstByte >= '0' && firstByte <= '9') {
+			JsonRpc::SendMessage(client, new Dictionary({
+				{ "jsonrpc", "2.0" },
+				{ "method", "icinga::Hello" },
+				{ "params", new Dictionary({
+					{ "version", (double)l_AppVersionInt },
+					{ "capabilities", (double)l_MyCapabilities }
+				}) }
+			}), yc);
+
+			client->async_flush(yc);
+
 			ctype = ClientJsonRpc;
 		} else {
 			ctype = ClientHttp;
@ -1613,6 +1645,19 @@ std::set<HttpServerConnection::Ptr> ApiListener::GetHttpClients() const

 Value ApiListener::HelloAPIHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params)
 {
+	if (origin) {
+		auto client (origin->FromClient);
+
+		if (client) {
+			auto endpoint (client->GetEndpoint());
+
+			if (endpoint) {
+				endpoint->SetIcingaVersion((double)params->Get("version"));
+				endpoint->SetCapabilities((double)params->Get("capabilities"));
+			}
+		}
+	}
+
 	return Empty;
 }

--- a/lib/remote/apilistener.hpp
+++ b/lib/remote/apilistener.hpp
@ -21,6 +21,7 @@
 #include <boost/asio/ip/tcp.hpp>
 #include <boost/asio/spawn.hpp>
 #include <boost/asio/ssl/context.hpp>
+#include <cstdint>
 #include <mutex>
 #include <set>

@ -39,6 +40,35 @@ struct ConfigDirInformation
 	Dictionary::Ptr Checksums;
 };

+/**
+ * If the version reported by icinga::Hello is not enough to tell whether
+ * the peer has a specific capability, add the latter to this bitmask.
+ *
+ * Note that due to the capability exchange via JSON-RPC and the state storage via JSON
+ * the bitmask numbers are stored in IEEE 754 64-bit floats.
+ * The latter have 53 digit bits which limit the bitmask.
+ * Not to run out of bits:
+ *
+ * Once all Icinga versions which don't have a specific capability are completely EOL,
+ * remove the respective capability checks and assume the peer has the capability.
+ * Once all Icinga versions which still check for the capability are completely EOL,
+ * remove the respective bit from icinga::Hello.
+ * Once all Icinga versions which still have the respective bit in icinga::Hello
+ * are completely EOL, remove the bit here.
+ * Once all Icinga versions which still have the respective bit here
+ * are completely EOL, feel free to re-use the bit.
+ *
+ * completely EOL = not supported, even if an important customer of us used it and
+ * not expected to appear in a multi-level cluster, e.g. a 4 level cluster with
+ * v2.11 -> v2.10 -> v2.9 -> v2.8 - v2.7 isn't here
+ *
+ * @ingroup remote
+ */
+enum class ApiCapabilities : uint_fast64_t
+{
+	ExecuteArbitraryCommand = 1u
+};
+
 /**
 * @ingroup remote
 */
--- a/lib/remote/endpoint.ti
+++ b/lib/remote/endpoint.ti
@ -1,6 +1,7 @@
 /* Icinga 2 | (c) 2012 Icinga GmbH | GPLv2+ */

 #include "base/configobject.hpp"
+#include <cstdint>

 library remote;

@ -21,6 +22,12 @@ class Endpoint : ConfigObject

 	[state] Timestamp local_log_position;
 	[state] Timestamp remote_log_position;
+	[state] "unsigned long" icinga_version {
+		default {{{ return 0; }}}
+	};
+	[state] uint_fast64_t capabilities {
+		default {{{ return 0; }}}
+	};

 	[no_user_modify] bool connecting;
 	[no_user_modify] bool syncing;
--- a/tools/mkclass/classcompiler.cpp
+++ b/tools/mkclass/classcompiler.cpp
@ -797,7 +797,7 @@ void ClassCompiler::HandleClass(const Klass& klass, const ClassDebugInfo&)
 					<< "{" << std::endl;

 				if (field.GetAccessor.empty() && !(field.Attributes & FANoStorage))
-					m_Impl << "\t" << "return m_" << field.GetFriendlyName() << ";" << std::endl;
+					m_Impl << "\t" << "return m_" << field.GetFriendlyName() << ".load();" << std::endl;
 				else
 					m_Impl << field.GetAccessor << std::endl;

@ -835,7 +835,7 @@ void ClassCompiler::HandleClass(const Klass& klass, const ClassDebugInfo&)

 					
 				if (field.SetAccessor.empty() && !(field.Attributes & FANoStorage))
-					m_Impl << "\t" << "m_" << field.GetFriendlyName() << " = value;" << std::endl;
+					m_Impl << "\t" << "m_" << field.GetFriendlyName() << ".store(value);" << std::endl;
 				else
 					m_Impl << field.SetAccessor << std::endl << std::endl;

@ -1044,7 +1044,7 @@ void ClassCompiler::HandleClass(const Klass& klass, const ClassDebugInfo&)
 			if (field.Attributes & FANoStorage)
 				continue;

-			m_Header << "\t" << field.Type.GetRealType() << " m_" << field.GetFriendlyName() << ";" << std::endl;
+			m_Header << "\tEventuallyAtomic<" << field.Type.GetRealType() << ">::type m_" << field.GetFriendlyName() << ";" << std::endl;
 		}
 		
 		/* signal */
@ -1431,6 +1431,7 @@ void ClassCompiler::CompileStream(const std::string& path, std::istream& input,
 		<< "#include \"base/type.hpp\"" << std::endl
 		<< "#include \"base/value.hpp\"" << std::endl
 		<< "#include \"base/array.hpp\"" << std::endl
+		<< "#include \"base/atomic.hpp\"" << std::endl
 		<< "#include \"base/dictionary.hpp\"" << std::endl
 		<< "#include <boost/signals2.hpp>" << std::endl << std::endl;