diff --git a/doc/16-upgrading-icinga-2.md b/doc/16-upgrading-icinga-2.md
index 39d35a5ab..67d79f4e2 100644
--- a/doc/16-upgrading-icinga-2.md
+++ b/doc/16-upgrading-icinga-2.md
@@ -96,7 +96,33 @@ feature with the `cipher_list` attribute.
In case that one of these ciphers is marked as insecure in the future,
please let us know with an issue on GitHub.
-### HA-aware Features
+### Cluster
+
+#### Config Sync
+
+2.11 overhauls the cluster config sync in many ways. This includes the
+following under the hood:
+
+- Synced configuration files are not immediately put into production, but left inside a stage.
+- Unsuccessful config validation never puts the config into production, additional logging and API states are available.
+- Zone directories which are not configured in zones.conf, are not included anymore on secondary master/satellites/clients.
+- Synced config change calculation use checksums instead of timestamps to trigger validation/reload. This is more safe, and the usage of timestamps is now deprecated.
+- Don't allow parallel cluster syncs to avoid race conditions with overridden files.
+- Deleted directories and files are now purged, previous versions had a bug.
+
+Whenever a newer child endpoint receives a configuration update without
+checksums, it will log a warning.
+
+```
+Received configuration update without checksums from parent endpoint satellite1. This behaviour is deprecated. Please upgrade the parent endpoint to 2.11+
+```
+
+This is a gentle reminder to upgrade the master and satellites first,
+prior to installing new clients/agents.
+
+Technical details are available in the [technical concepts](19-technical-concepts.md#technical-concepts-cluster-config-sync) chapter.
+
+#### HA-aware Features
v2.11 introduces additional HA functionality similar to the DB IDO feature.
This enables the feature being active only on one endpoint while the other
@@ -182,7 +208,7 @@ constant in [constants.conf](04-configuring-icinga-2.md#constants-conf) instead.
### REST API
-#### Actions
+#### Actions
The [schedule-downtime](12-icinga2-api.md#icinga2-api-actions-schedule-downtime-host-all-services)
action supports the `all_services` parameter for Host types. Defaults to false.
diff --git a/doc/19-technical-concepts.md b/doc/19-technical-concepts.md
index e6483132a..12ed44f4b 100644
--- a/doc/19-technical-concepts.md
+++ b/doc/19-technical-concepts.md
@@ -813,6 +813,188 @@ Icinga 2 v2.9+ adds more performance metrics for these values:
* `sum_bytes_sent_per_second` and `sum_bytes_received_per_second`
+### Config Sync
+
+The visible feature for the user is to put configuration files in `/etc/icinga2/zones.d/`
+and have them synced automatically to all involved zones and endpoints.
+
+This not only includes host and service objects being checked
+in a satellite zone, but also additional config objects such as
+commands, groups, timeperiods and also templates.
+
+Additional thoughts and complexity added:
+
+- Putting files into zone directory names removes the burden to set the `zone` attribute on each object in this directory. This is done automatically by the config compiler.
+- Inclusion of `zones.d` happens automatically, the user shouldn't be bothered about this.
+- Before the REST API was created, only static configuration files in `/etc/icinga2/zones.d` existed. With the addition of config packages, additional `zones.d` targets must be registered (e.g. used by the Director)
+- Only one config master is allowed. This one identifies itself with configuration files in `/etc/icinga2/zones.d`. This is not necessarily the zone master seen in the debug logs, that one is important for message routing internally.
+- Objects and templates which cannot be bound into a specific zone (e.g. hosts in the satellite zone) must be made available "globally".
+- Users must be able to deny the synchronisation of specific zones, e.g. for security reasons.
+
+#### Config Sync: Config Master
+
+All zones must be configured and included in the `zones.conf` config file beforehand.
+The zone names are the identifier for the directories underneath the `/etc/icinga2/zones.d`
+directory. If a zone is not configured, it will not be included in the config sync - keep this
+in mind for troubleshooting.
+
+When the config master starts, the content of `/etc/icinga2/zones.d` is automatically
+included. There's no need for an additional entry in `icinga2.conf` like `conf.d`.
+You can verify this by running the config validation on debug level:
+
+```
+icinga2 daemon -C -x debug | grep 'zones.d'
+
+[2019-06-19 15:16:19 +0200] notice/ConfigCompiler: Compiling config file: /etc/icinga2/zones.d/global-templates/commands.conf
+```
+
+Once the config validation succeeds, the startup routine for the daemon
+copies the files into the "production" directory in `/var/lib/icinga2/api/zones`.
+This directory is used for all endpoints where Icinga stores the received configuration.
+With the exception of the config master retrieving this from `/etc/icinga2/zones.d` instead.
+
+These operations are logged for better visibility.
+
+```
+[2019-06-19 15:26:38 +0200] information/ApiListener: Copying 1 zone configuration files for zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'.
+[2019-06-19 15:26:38 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/commands.conf
+```
+
+The master is finished at this point. Depending on the cluster configuration,
+the next iteration is a connected endpoint after successful TLS handshake and certificate
+authentication.
+
+It calls `SendConfigUpdate(client)` which sends the [config::Update](19-technical-concepts.md#technical-concepts-json-rpc-messages-config-update)
+JSON-RPC message including all required zones and their configuration file content.
+
+
+#### Config Sync: Receive Config
+
+The secondary master endpoint and endpoints in a child zone will be connected to the config
+master. The endpoint receives the [config::Update](19-technical-concepts.md#technical-concepts-json-rpc-messages-config-update)
+JSON-RPC message and processes the content in `ConfigUpdateHandler()`. This method checks
+whether config should be accepted. In addition to that, it locks a local mutex to avoid race conditions
+with multiple syncs in parallel.
+
+After that, the received configuration content is analysed.
+
+> **Note**
+>
+> The cluster design allows that satellite endpoints may connect to the secondary master first.
+> There is no immediate need to always connect to the config master first, especially since
+> the satellite endpoints don't know that.
+>
+> The secondary master not only stores the master zone config files, but also all child zones.
+> This is also the case for any HA enabled zone with more than one endpoint.
+
+
+2.11 puts the received configuration files into a staging directory in
+`/var/lib/icinga2/api/zones-stage`. Previous versions directly wrote the
+files into production which could have led to broken configuration on the
+next manual restart.
+
+```
+[2019-06-19 16:08:29 +0200] information/ApiListener: New client connection for identity 'master1' to [127.0.0.1]:5665
+[2019-06-19 16:08:30 +0200] information/ApiListener: Applying config update from endpoint 'master1' of zone 'master'.
+[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master1'. Comparing the checksums.
+[2019-06-19 16:08:30 +0200] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/agent//_etc/host.conf' for zone 'agent'.
+[2019-06-19 16:08:30 +0200] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/agent' (176 Bytes).
+[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration for zone 'master' from endpoint 'master1'. Comparing the checksums.
+[2019-06-19 16:08:30 +0200] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/master' (17 Bytes).
+[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration from endpoint 'master1' is different to production, triggering validation and reload.
+```
+
+It then validates the received configuration in its own config stage. There is
+an parameter override in place which disables the automatic inclusion of the production
+config in `/var/lib/icinga2/api/zones`.
+
+Once completed, the reload is triggered. This follows the same configurable timeout
+as with the global reload.
+
+```
+[2019-06-19 16:52:26 +0200] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
+[2019-06-19 16:52:27 +0200] information/Application: Got reload command: Started new instance with PID '19945' (timeout is 300s).
+[2019-06-19 16:52:28 +0200] information/Application: Reload requested, letting new process take over.
+```
+
+Whenever the staged configuration validation fails, Icinga logs this including a reference
+to the startup log file which includes additional errors.
+
+```
+[2019-06-19 15:45:27 +0200] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Aborting. Logs: '/var/lib/icinga2/api/zones-stage//startup.log'
+```
+
+
+#### Config Sync: Changes and Reload
+
+Whenever a new configuration is received, it is validated and upon success, the
+daemon automatically reloads. While the daemon continues with checks, the reload
+cannot hand over open TCP connections. That being said, reloading the daemon everytime
+a configuration is synchronized would lead into many not connected endpoints.
+
+Therefore the cluster config sync checks whether the configuration files actually
+changed, and will only trigger a reload when such a change happened.
+
+2.11 calculates a checksum from each file content and compares this to the
+production configuration. Previous versions used additional metadata with timestamps from
+files which sometimes led to problems with asynchronous dates.
+
+> **Note**
+>
+> For compatibility reasons, the timestamp metadata algorithm is still intact, e.g.
+> when the client is 2.11 already, but the parent endpoint is still on 2.10.
+
+Icinga logs a warning when this happens.
+
+```
+Received configuration update without checksums from parent endpoint satellite1. This behaviour is deprecated. Please upgrade the parent endpoint to 2.11+
+```
+
+
+The debug log provides more details on the actual checksums and checks. Future output
+may change, use this solely for troubleshooting and debugging whenever the cluster
+config sync fails.
+
+```
+[2019-06-19 16:13:16 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master1'. Comparing the checksums.
+[2019-06-19 16:13:16 +0200] debug/ApiListener: Checking for config change between stage and production. Old (3): '{"/.checksums":"7ede1276a9a32019c1412a52779804a976e163943e268ec4066e6b6ec4d15d73","/.timestamp":"ec4354b0eca455f7c2ca386fddf5b9ea810d826d402b3b6ac56ba63b55c2892c","/_etc/host.conf":"35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44"}' vs. new (3): '{"/.checksums":"84a586435d732327e2152e7c9b6d85a340cc917b89ae30972042f3dc344ea7cf","/.timestamp":"0fd6facf35e49ab1b2a161872fa7ad794564eba08624373d99d31c32a7a4c7d3","/_etc/host.conf":"0d62075e89be14088de1979644b40f33a8f185fcb4bb6ff1f7da2f63c7723fcb"}'.
+[2019-06-19 16:13:16 +0200] debug/ApiListener: Checking /_etc/host.conf for checksum: 35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44
+[2019-06-19 16:13:16 +0200] debug/ApiListener: Path '/_etc/host.conf' doesn't match old checksum '0d62075e89be14088de1979644b40f33a8f185fcb4bb6ff1f7da2f63c7723fcb' with new checksum '35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44'.
+```
+
+
+#### Config Sync: Trust
+
+The config sync follows the "top down" approach, where the master endpoint in the master
+zone is allowed to synchronize configuration to the child zone, e.g. the satellite zone.
+
+Endpoints in the same zone, e.g. a secondary master, receive configuration for the same
+zone and all child zones.
+
+Endpoints in the satellite zone trust the parent zone, and will accept the pushed
+configuration via JSON-RPC cluster messages. By default, this is disabled and must
+be enabled with the `accept_config` attribute in the ApiListener feature (manually or with CLI
+helpers).
+
+The satellite zone will not only accept zone configuration for its own zone, but also
+all configured child zones. That is why it is important to configure the zone hierarchy
+on the satellite as well.
+
+Child zones are not allowed to sync configuration up to the parent zone. Each Icinga instance
+evaluates this in startup and knows on endpoint connect which config zones need to be synced.
+
+
+Global zones have a special trust relationship: They are synced to all child zones, be it
+a satellite zone or client zone. Since checkable objects such as a Host or a Service object
+must have only one endpoint as authority, they cannot be put into a global zone (denied by
+the config compiler).
+
+Apply rules and templates are allowed, since they are evaluated in the endpoint which received
+the synced configuration. Keep in mind that there may be differences on the master and the satellite
+when e.g. hostgroup membership is used for assign where expressions, but the groups are only
+available on the master.
+
+
## TLS Network IO
### TLS Connection Handling
diff --git a/lib/cli/daemonutility.cpp b/lib/cli/daemonutility.cpp
index b586544ee..d5bb2cf1e 100644
--- a/lib/cli/daemonutility.cpp
+++ b/lib/cli/daemonutility.cpp
@@ -100,6 +100,10 @@ static void IncludePackage(const String& packagePath, bool& success)
bool DaemonUtility::ValidateConfigFiles(const std::vector& configs, const String& objectsFile)
{
bool success;
+
+ Namespace::Ptr systemNS = ScriptGlobal::Get("System");
+ VERIFY(systemNS);
+
if (!objectsFile.IsEmpty())
ConfigCompilerContext::GetInstance()->OpenObjectsFile(objectsFile);
@@ -122,12 +126,15 @@ bool DaemonUtility::ValidateConfigFiles(const std::vector& configs,
* unfortunately moving it there is somewhat non-trivial. */
success = true;
- String zonesEtcDir = Configuration::ZonesDir;
- if (!zonesEtcDir.IsEmpty() && Utility::PathExists(zonesEtcDir))
- Utility::Glob(zonesEtcDir + "/*", std::bind(&IncludeZoneDirRecursive, _1, "_etc", std::ref(success)), GlobDirectory);
+ /* Only load zone directory if we're not in staging validation. */
+ if (!systemNS->Contains("ZonesStageVarDir")) {
+ String zonesEtcDir = Configuration::ZonesDir;
+ if (!zonesEtcDir.IsEmpty() && Utility::PathExists(zonesEtcDir))
+ Utility::Glob(zonesEtcDir + "/*", std::bind(&IncludeZoneDirRecursive, _1, "_etc", std::ref(success)), GlobDirectory);
- if (!success)
- return false;
+ if (!success)
+ return false;
+ }
/* Load package config files - they may contain additional zones which
* are authoritative on this node and are checked in HasZoneConfigAuthority(). */
@@ -138,17 +145,24 @@ bool DaemonUtility::ValidateConfigFiles(const std::vector& configs,
if (!success)
return false;
- /* Load cluster synchronized configuration files */
+ /* Load cluster synchronized configuration files. This can be overridden for staged sync validations. */
String zonesVarDir = Configuration::DataDir + "/api/zones";
+
+ /* Cluster config sync stage validation needs this. */
+ if (systemNS->Contains("ZonesStageVarDir")) {
+ zonesVarDir = systemNS->Get("ZonesStageVarDir");
+
+ Log(LogNotice, "DaemonUtility")
+ << "Overriding zones var directory with '" << zonesVarDir << "' for cluster config sync staging.";
+ }
+
+
if (Utility::PathExists(zonesVarDir))
Utility::Glob(zonesVarDir + "/*", std::bind(&IncludeNonLocalZone, _1, "_cluster", std::ref(success)), GlobDirectory);
if (!success)
return false;
- Namespace::Ptr systemNS = ScriptGlobal::Get("System");
- VERIFY(systemNS);
-
/* This is initialized inside the IcingaApplication class. */
Value vAppType;
VERIFY(systemNS->Get("ApplicationType", &vAppType));
diff --git a/lib/methods/icingachecktask.cpp b/lib/methods/icingachecktask.cpp
index 52213889e..fa598d487 100644
--- a/lib/methods/icingachecktask.cpp
+++ b/lib/methods/icingachecktask.cpp
@@ -8,6 +8,7 @@
#include "icinga/icingaapplication.hpp"
#include "icinga/clusterevents.hpp"
#include "icinga/checkable.hpp"
+#include "remote/apilistener.hpp"
#include "base/application.hpp"
#include "base/objectlock.hpp"
#include "base/utility.hpp"
@@ -157,6 +158,20 @@ void IcingaCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
cr->SetState(ServiceWarning);
}
+ /* Indicate a warning when the last synced config caused a stage validation error. */
+ ApiListener::Ptr listener = ApiListener::GetInstance();
+
+ if (listener) {
+ Dictionary::Ptr validationResult = listener->GetLastFailedZonesStageValidation();
+
+ if (validationResult) {
+ output += "; Last zone sync stage validation failed at "
+ + Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", validationResult->Get("ts"));
+
+ cr->SetState(ServiceWarning);
+ }
+ }
+
/* Extract the version number of the running Icinga2 instance.
* We assume that appVersion will allways be something like 'v2.10.1-8-gaebe6da' and we want to extract '2.10.1'.
*/
diff --git a/lib/remote/CMakeLists.txt b/lib/remote/CMakeLists.txt
index da0006aa1..2c5a0326a 100644
--- a/lib/remote/CMakeLists.txt
+++ b/lib/remote/CMakeLists.txt
@@ -59,6 +59,7 @@ set_target_properties (
#install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/api\")")
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/api/log\")")
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/api/zones\")")
+install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/api/zones-stage\")")
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/certs\")")
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/certificate-requests\")")
diff --git a/lib/remote/apilistener-filesync.cpp b/lib/remote/apilistener-filesync.cpp
index 411028630..f3d5a5d48 100644
--- a/lib/remote/apilistener-filesync.cpp
+++ b/lib/remote/apilistener-filesync.cpp
@@ -3,9 +3,12 @@
#include "remote/apilistener.hpp"
#include "remote/apifunction.hpp"
#include "config/configcompiler.hpp"
+#include "base/tlsutility.hpp"
+#include "base/json.hpp"
#include "base/configtype.hpp"
#include "base/logger.hpp"
#include "base/convert.hpp"
+#include "base/application.hpp"
#include "base/exception.hpp"
#include "base/utility.hpp"
#include
@@ -17,233 +20,200 @@ REGISTER_APIFUNCTION(Update, config, &ApiListener::ConfigUpdateHandler);
boost::mutex ApiListener::m_ConfigSyncStageLock;
-void ApiListener::ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file)
-{
- CONTEXT("Creating config update for file '" + file + "'");
-
- Log(LogNotice, "ApiListener")
- << "Creating config update for file '" << file << "'.";
-
- std::ifstream fp(file.CStr(), std::ifstream::binary);
- if (!fp)
- return;
-
- String content((std::istreambuf_iterator(fp)), std::istreambuf_iterator());
-
- Dictionary::Ptr update;
-
- if (Utility::Match("*.conf", file))
- update = config.UpdateV1;
- else
- update = config.UpdateV2;
-
- update->Set(file.SubStr(path.GetLength()), content);
-}
-
-Dictionary::Ptr ApiListener::MergeConfigUpdate(const ConfigDirInformation& config)
-{
- Dictionary::Ptr result = new Dictionary();
-
- if (config.UpdateV1)
- config.UpdateV1->CopyTo(result);
-
- if (config.UpdateV2)
- config.UpdateV2->CopyTo(result);
-
- return result;
-}
-
-ConfigDirInformation ApiListener::LoadConfigDir(const String& dir)
-{
- ConfigDirInformation config;
- config.UpdateV1 = new Dictionary();
- config.UpdateV2 = new Dictionary();
- Utility::GlobRecursive(dir, "*", std::bind(&ApiListener::ConfigGlobHandler, std::ref(config), dir, _1), GlobFile);
- return config;
-}
-
-bool ApiListener::UpdateConfigDir(const ConfigDirInformation& oldConfigInfo, const ConfigDirInformation& newConfigInfo, const String& configDir, bool authoritative)
-{
- bool configChange = false;
-
- Dictionary::Ptr oldConfig = MergeConfigUpdate(oldConfigInfo);
- Dictionary::Ptr newConfig = MergeConfigUpdate(newConfigInfo);
-
- double oldTimestamp;
-
- if (!oldConfig->Contains("/.timestamp"))
- oldTimestamp = 0;
- else
- oldTimestamp = oldConfig->Get("/.timestamp");
-
- double newTimestamp;
-
- if (!newConfig->Contains("/.timestamp"))
- newTimestamp = Utility::GetTime();
- else
- newTimestamp = newConfig->Get("/.timestamp");
-
- /* skip update if our configuration files are more recent */
- if (oldTimestamp >= newTimestamp) {
- Log(LogNotice, "ApiListener")
- << "Our configuration is more recent than the received configuration update."
- << " Ignoring configuration file update for path '" << configDir << "'. Current timestamp '"
- << Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", oldTimestamp) << "' ("
- << std::fixed << std::setprecision(6) << oldTimestamp
- << ") >= received timestamp '"
- << Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", newTimestamp) << "' ("
- << newTimestamp << ").";
- return false;
- }
-
- size_t numBytes = 0;
-
- {
- ObjectLock olock(newConfig);
- for (const Dictionary::Pair& kv : newConfig) {
- if (oldConfig->Get(kv.first) != kv.second) {
- if (!Utility::Match("*/.timestamp", kv.first))
- configChange = true;
-
- String path = configDir + "/" + kv.first;
- Log(LogInformation, "ApiListener")
- << "Updating configuration file: " << path;
-
- /* Sync string content only. */
- String content = kv.second;
-
- /* Generate a directory tree (zones/1/2/3 might not exist yet). */
- Utility::MkDirP(Utility::DirName(path), 0755);
- std::ofstream fp(path.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
- fp << content;
- fp.close();
-
- numBytes += content.GetLength();
- }
- }
- }
-
- Log(LogInformation, "ApiListener")
- << "Applying configuration file update for path '" << configDir << "' (" << numBytes << " Bytes). Received timestamp '"
- << Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", newTimestamp) << "' ("
- << std::fixed << std::setprecision(6) << newTimestamp
- << "), Current timestamp '"
- << Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", oldTimestamp) << "' ("
- << oldTimestamp << ").";
-
- ObjectLock xlock(oldConfig);
- for (const Dictionary::Pair& kv : oldConfig) {
- if (!newConfig->Contains(kv.first)) {
- configChange = true;
-
- String path = configDir + "/" + kv.first;
- (void) unlink(path.CStr());
- }
- }
-
- String tsPath = configDir + "/.timestamp";
- if (!Utility::PathExists(tsPath)) {
- std::ofstream fp(tsPath.CStr(), std::ofstream::out | std::ostream::trunc);
- fp << std::fixed << newTimestamp;
- fp.close();
- }
-
- if (authoritative) {
- String authPath = configDir + "/.authoritative";
- if (!Utility::PathExists(authPath)) {
- std::ofstream fp(authPath.CStr(), std::ofstream::out | std::ostream::trunc);
- fp.close();
- }
- }
-
- return configChange;
-}
-
-void ApiListener::SyncZoneDir(const Zone::Ptr& zone) const
-{
- ConfigDirInformation newConfigInfo;
- newConfigInfo.UpdateV1 = new Dictionary();
- newConfigInfo.UpdateV2 = new Dictionary();
-
- for (const ZoneFragment& zf : ConfigCompiler::GetZoneDirs(zone->GetName())) {
- ConfigDirInformation newConfigPart = LoadConfigDir(zf.Path);
-
- {
- ObjectLock olock(newConfigPart.UpdateV1);
- for (const Dictionary::Pair& kv : newConfigPart.UpdateV1) {
- newConfigInfo.UpdateV1->Set("/" + zf.Tag + kv.first, kv.second);
- }
- }
-
- {
- ObjectLock olock(newConfigPart.UpdateV2);
- for (const Dictionary::Pair& kv : newConfigPart.UpdateV2) {
- newConfigInfo.UpdateV2->Set("/" + zf.Tag + kv.first, kv.second);
- }
- }
- }
-
- int sumUpdates = newConfigInfo.UpdateV1->GetLength() + newConfigInfo.UpdateV2->GetLength();
-
- if (sumUpdates == 0)
- return;
-
- String oldDir = Configuration::DataDir + "/api/zones/" + zone->GetName();
-
- Log(LogInformation, "ApiListener")
- << "Copying " << sumUpdates << " zone configuration files for zone '" << zone->GetName() << "' to '" << oldDir << "'.";
-
- Utility::MkDirP(oldDir, 0700);
-
- ConfigDirInformation oldConfigInfo = LoadConfigDir(oldDir);
-
- UpdateConfigDir(oldConfigInfo, newConfigInfo, oldDir, true);
-}
-
-void ApiListener::SyncZoneDirs() const
+/**
+ * Entrypoint for updating all authoritative configs from /etc/zones.d, packages, etc.
+ * into var/lib/icinga2/api/zones
+ */
+void ApiListener::SyncLocalZoneDirs() const
{
for (const Zone::Ptr& zone : ConfigType::GetObjectsByType()) {
try {
- SyncZoneDir(zone);
+ SyncLocalZoneDir(zone);
} catch (const std::exception&) {
continue;
}
}
}
+/**
+ * Sync a zone directory where we have an authoritative copy (zones.d, packages, etc.)
+ *
+ * This function collects the registered zone config dirs from
+ * the config compiler and reads the file content into the config
+ * information structure.
+ *
+ * Returns early when there are no updates.
+ *
+ * @param zone Pointer to the zone object being synced.
+ */
+void ApiListener::SyncLocalZoneDir(const Zone::Ptr& zone) const
+{
+ if (!zone)
+ return;
+
+ ConfigDirInformation newConfigInfo;
+ newConfigInfo.UpdateV1 = new Dictionary();
+ newConfigInfo.UpdateV2 = new Dictionary();
+ newConfigInfo.Checksums = new Dictionary();
+
+ String zoneName = zone->GetName();
+
+ // Load registered zone paths, e.g. '_etc', '_api' and user packages.
+ for (const ZoneFragment& zf : ConfigCompiler::GetZoneDirs(zoneName)) {
+ ConfigDirInformation newConfigPart = LoadConfigDir(zf.Path);
+
+ // Config files '*.conf'.
+ {
+ ObjectLock olock(newConfigPart.UpdateV1);
+ for (const Dictionary::Pair& kv : newConfigPart.UpdateV1) {
+ String path = "/" + zf.Tag + kv.first;
+
+ newConfigInfo.UpdateV1->Set(path, kv.second);
+ newConfigInfo.Checksums->Set(path, GetChecksum(kv.second));
+ }
+ }
+
+ // Meta files.
+ {
+ ObjectLock olock(newConfigPart.UpdateV2);
+ for (const Dictionary::Pair& kv : newConfigPart.UpdateV2) {
+ String path = "/" + zf.Tag + kv.first;
+
+ newConfigInfo.UpdateV2->Set(path, kv.second);
+ newConfigInfo.Checksums->Set(path, GetChecksum(kv.second));
+ }
+ }
+ }
+
+ size_t sumUpdates = newConfigInfo.UpdateV1->GetLength() + newConfigInfo.UpdateV2->GetLength();
+
+ // Return early if there are no updates.
+ if (sumUpdates == 0)
+ return;
+
+ String productionZonesDir = GetApiZonesDir() + zoneName;
+
+ Log(LogInformation, "ApiListener")
+ << "Copying " << sumUpdates << " zone configuration files for zone '" << zoneName << "' to '" << productionZonesDir << "'.";
+
+ // Purge files to allow deletion via zones.d.
+ if (Utility::PathExists(productionZonesDir))
+ Utility::RemoveDirRecursive(productionZonesDir);
+
+ Utility::MkDirP(productionZonesDir, 0700);
+
+ // Copy content and add additional meta data.
+ size_t numBytes = 0;
+
+ /* Note: We cannot simply copy directories here.
+ *
+ * Zone directories are registered from everywhere and we already
+ * have read their content into memory with LoadConfigDir().
+ */
+ Dictionary::Ptr newConfig = MergeConfigUpdate(newConfigInfo);
+
+ {
+ ObjectLock olock(newConfig);
+
+ for (const Dictionary::Pair& kv : newConfig) {
+ String dst = productionZonesDir + "/" + kv.first;
+
+ Utility::MkDirP(Utility::DirName(dst), 0755);
+
+ Log(LogInformation, "ApiListener")
+ << "Updating configuration file: " << dst;
+
+ String content = kv.second;
+
+ std::ofstream fp(dst.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
+
+ fp << content;
+ fp.close();
+
+ numBytes += content.GetLength();
+ }
+ }
+
+ // Additional metadata.
+ String tsPath = productionZonesDir + "/.timestamp";
+
+ if (!Utility::PathExists(tsPath)) {
+ std::ofstream fp(tsPath.CStr(), std::ofstream::out | std::ostream::trunc);
+
+ fp << std::fixed << Utility::GetTime();
+ fp.close();
+ }
+
+ String authPath = productionZonesDir + "/.authoritative";
+
+ if (!Utility::PathExists(authPath)) {
+ std::ofstream fp(authPath.CStr(), std::ofstream::out | std::ostream::trunc);
+ fp.close();
+ }
+
+ // Checksums.
+ String checksumsPath = productionZonesDir + "/.checksums";
+
+ if (Utility::PathExists(checksumsPath))
+ Utility::Remove(checksumsPath);
+
+ std::ofstream fp(checksumsPath.CStr(), std::ofstream::out | std::ostream::trunc);
+
+ fp << std::fixed << JsonEncode(newConfigInfo.Checksums);
+ fp.close();
+
+ Log(LogNotice, "ApiListener")
+ << "Updated meta data for cluster config sync. Checksum: '" << checksumsPath
+ << "', timestamp: '" << tsPath << "', auth: '" << authPath << "'.";
+}
+
+/**
+ * Entrypoint for sending a file based config update to a cluster client.
+ * This includes security checks for zone relations.
+ * Loads the zone config files where this client belongs to
+ * and sends the 'config::Update' JSON-RPC message.
+ *
+ * @param aclient Connected JSON-RPC client.
+ */
void ApiListener::SendConfigUpdate(const JsonRpcConnection::Ptr& aclient)
{
Endpoint::Ptr endpoint = aclient->GetEndpoint();
ASSERT(endpoint);
- Zone::Ptr azone = endpoint->GetZone();
- Zone::Ptr lzone = Zone::GetLocalZone();
+ Zone::Ptr clientZone = endpoint->GetZone();
+ Zone::Ptr localZone = Zone::GetLocalZone();
- /* don't try to send config updates to our master */
- if (!azone->IsChildOf(lzone))
+ // Don't send config updates to parent zones
+ if (!clientZone->IsChildOf(localZone))
return;
Dictionary::Ptr configUpdateV1 = new Dictionary();
Dictionary::Ptr configUpdateV2 = new Dictionary();
+ Dictionary::Ptr configUpdateChecksums = new Dictionary(); // new since 2.11
- String zonesDir = Configuration::DataDir + "/api/zones";
+ String zonesDir = GetApiZonesDir();
for (const Zone::Ptr& zone : ConfigType::GetObjectsByType()) {
- String zoneDir = zonesDir + "/" + zone->GetName();
+ String zoneName = zone->GetName();
+ String zoneDir = zonesDir + zoneName;
- if (!zone->IsChildOf(azone) && !zone->IsGlobal())
+ // Only sync child and global zones.
+ if (!zone->IsChildOf(clientZone) && !zone->IsGlobal())
continue;
+ // Zone was configured, but there's no configuration directory.
if (!Utility::PathExists(zoneDir))
continue;
Log(LogInformation, "ApiListener")
<< "Syncing configuration files for " << (zone->IsGlobal() ? "global " : "")
- << "zone '" << zone->GetName() << "' to endpoint '" << endpoint->GetName() << "'.";
+ << "zone '" << zoneName << "' to endpoint '" << endpoint->GetName() << "'.";
- ConfigDirInformation config = LoadConfigDir(zonesDir + "/" + zone->GetName());
- configUpdateV1->Set(zone->GetName(), config.UpdateV1);
- configUpdateV2->Set(zone->GetName(), config.UpdateV2);
+ ConfigDirInformation config = LoadConfigDir(zoneDir);
+
+ configUpdateV1->Set(zoneName, config.UpdateV1);
+ configUpdateV2->Set(zoneName, config.UpdateV2);
+ configUpdateChecksums->Set(zoneName, config.Checksums); // new since 2.11
}
Dictionary::Ptr message = new Dictionary({
@@ -251,15 +221,29 @@ void ApiListener::SendConfigUpdate(const JsonRpcConnection::Ptr& aclient)
{ "method", "config::Update" },
{ "params", new Dictionary({
{ "update", configUpdateV1 },
- { "update_v2", configUpdateV2 }
+ { "update_v2", configUpdateV2 }, // Since 2.4.2.
+ { "checksums", configUpdateChecksums } // Since 2.11.0.
}) }
});
aclient->SendMessage(message);
}
+/**
+ * Registered handler when a new config::Update message is received.
+ *
+ * Checks destination and permissions first, locks the transaction and analyses the update.
+ * The newly received configuration is not copied to production immediately,
+ * but into the staging directory first.
+ * Last, the async validation and restart is triggered.
+ *
+ * @param origin Where this message came from.
+ * @param params Message parameters including the config updates.
+ * @returns Empty, required by the interface.
+ */
Value ApiListener::ConfigUpdateHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params)
{
+ // Verify permissions and trust relationship.
if (!origin->FromClient->GetEndpoint() || (origin->FromZone && !Zone::GetLocalZone()->IsChildOf(origin->FromZone)))
return Empty;
@@ -281,52 +265,526 @@ Value ApiListener::ConfigUpdateHandler(const MessageOrigin::Ptr& origin, const D
*/
boost::mutex::scoped_lock lock(m_ConfigSyncStageLock);
- Log(LogInformation, "ApiListener")
- << "Applying config update from endpoint '" << origin->FromClient->GetEndpoint()->GetName()
- << "' of zone '" << GetFromZoneName(origin->FromZone) << "'.";
+ String apiZonesStageDir = GetApiZonesStageDir();
+ String fromEndpointName = origin->FromClient->GetEndpoint()->GetName();
+ String fromZoneName = GetFromZoneName(origin->FromZone);
+ Log(LogInformation, "ApiListener")
+ << "Applying config update from endpoint '" << fromEndpointName
+ << "' of zone '" << fromZoneName << "'.";
+
+ // Config files.
Dictionary::Ptr updateV1 = params->Get("update");
+ // Meta data files: .timestamp, etc.
Dictionary::Ptr updateV2 = params->Get("update_v2");
+ // New since 2.11.0.
+ Dictionary::Ptr checksums;
+
+ if (params->Contains("checksums"))
+ checksums = params->Get("checksums");
+
bool configChange = false;
+ // Keep track of the relative config paths for later validation and copying. TODO: Find a better algorithm.
+ std::vector relativePaths;
+
+ /*
+ * We can and must safely purge the staging directory, as the difference is taken between
+ * runtime production config and newly received configuration.
+ * This is needed to not mix deleted/changed content between received and stage
+ * config.
+ */
+ if (Utility::PathExists(apiZonesStageDir))
+ Utility::RemoveDirRecursive(apiZonesStageDir);
+
+ Utility::MkDirP(apiZonesStageDir, 0700);
+
+ // Analyse and process the update.
ObjectLock olock(updateV1);
+
for (const Dictionary::Pair& kv : updateV1) {
- Zone::Ptr zone = Zone::GetByName(kv.first);
+
+ // Check for the configured zones.
+ String zoneName = kv.first;
+ Zone::Ptr zone = Zone::GetByName(zoneName);
if (!zone) {
Log(LogWarning, "ApiListener")
- << "Ignoring config update for unknown zone '" << kv.first << "'.";
+ << "Ignoring config update from endpoint '" << fromEndpointName
+ << "' for unknown zone '" << zoneName << "'.";
+
continue;
}
- if (ConfigCompiler::HasZoneConfigAuthority(kv.first)) {
- Log(LogWarning, "ApiListener")
- << "Ignoring config update for zone '" << kv.first << "' because we have an authoritative version of the zone's config.";
+ // Ignore updates where we have an authoritive copy in etc/zones.d, packages, etc.
+ if (ConfigCompiler::HasZoneConfigAuthority(zoneName)) {
+ Log(LogInformation, "ApiListener")
+ << "Ignoring config update from endpoint '" << fromEndpointName
+ << "' for zone '" << zoneName << "' because we have an authoritative version of the zone's config.";
+
continue;
}
- String oldDir = Configuration::DataDir + "/api/zones/" + zone->GetName();
+ // Put the received configuration into our stage directory.
+ String productionConfigZoneDir = GetApiZonesDir() + zoneName;
+ String stageConfigZoneDir = GetApiZonesStageDir() + zoneName;
- Utility::MkDirP(oldDir, 0700);
+ Utility::MkDirP(productionConfigZoneDir, 0700);
+ Utility::MkDirP(stageConfigZoneDir, 0700);
+ // Merge the config information.
ConfigDirInformation newConfigInfo;
newConfigInfo.UpdateV1 = kv.second;
+ // Load metadata.
if (updateV2)
newConfigInfo.UpdateV2 = updateV2->Get(kv.first);
- Dictionary::Ptr newConfig = kv.second;
- ConfigDirInformation oldConfigInfo = LoadConfigDir(oldDir);
+ // Load checksums. New since 2.11.
+ if (checksums)
+ newConfigInfo.Checksums = checksums->Get(kv.first);
- if (UpdateConfigDir(oldConfigInfo, newConfigInfo, oldDir, false))
- configChange = true;
+ // Load the current production config details.
+ ConfigDirInformation productionConfigInfo = LoadConfigDir(productionConfigZoneDir);
+
+ // Merge updateV1 and updateV2
+ Dictionary::Ptr productionConfig = MergeConfigUpdate(productionConfigInfo);
+ Dictionary::Ptr newConfig = MergeConfigUpdate(newConfigInfo);
+
+ /* If we have received 'checksums' via cluster message, go for it.
+ * Otherwise do the old timestamp dance for versions < 2.11.
+ */
+ if (checksums) {
+ Log(LogInformation, "ApiListener")
+ << "Received configuration for zone '" << zoneName << "' from endpoint '"
+ << fromEndpointName << "'. Comparing the checksums.";
+
+ // TODO: Do this earlier in hello-handshakes?
+ if (CheckConfigChange(productionConfigInfo, newConfigInfo))
+ configChange = true;
+
+ } else {
+ /* Fallback to timestamp handling when the parent endpoint didn't send checks.
+ * This can happen when the satellite is 2.11 and the master is 2.10.
+ *
+ * TODO: Deprecate and remove this behaviour in 2.13+.
+ */
+
+ Log(LogWarning, "ApiListener")
+ << "Received configuration update without checksums from parent endpoint "
+ << fromEndpointName << ". This behaviour is deprecated. Please upgrade the parent endpoint to 2.11+";
+
+ double productionTimestamp;
+
+ if (!productionConfig->Contains("/.timestamp"))
+ productionTimestamp = 0;
+ else
+ productionTimestamp = productionConfig->Get("/.timestamp");
+
+ double newTimestamp;
+
+ if (!newConfig->Contains("/.timestamp"))
+ newTimestamp = Utility::GetTime();
+ else
+ newTimestamp = newConfig->Get("/.timestamp");
+
+ // Skip update if our configuration files are more recent.
+ if (productionTimestamp >= newTimestamp) {
+
+ Log(LogInformation, "ApiListener")
+ << "Our configuration is more recent than the received configuration update."
+ << " Ignoring configuration file update for path '" << stageConfigZoneDir << "'. Current timestamp '"
+ << Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", productionTimestamp) << "' ("
+ << std::fixed << std::setprecision(6) << productionTimestamp
+ << ") >= received timestamp '"
+ << Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", newTimestamp) << "' ("
+ << newTimestamp << ").";
+
+ } else {
+ configChange = true;
+ }
+
+ // Keep another hack when there's a timestamp file missing.
+ {
+ ObjectLock olock(newConfig);
+
+ for (const Dictionary::Pair &kv : newConfig) {
+
+ // This is super expensive with a string content comparison.
+ if (productionConfig->Get(kv.first) != kv.second) {
+ if (!Utility::Match("*/.timestamp", kv.first))
+ configChange = true;
+ }
+ }
+ }
+
+ // Update the .timestamp file.
+ String tsPath = stageConfigZoneDir + "/.timestamp";
+ if (!Utility::PathExists(tsPath)) {
+ std::ofstream fp(tsPath.CStr(), std::ofstream::out | std::ostream::trunc);
+ fp << std::fixed << newTimestamp;
+ fp.close();
+ }
+ }
+
+ // Dump the received configuration for this zone into the stage directory.
+ size_t numBytes = 0;
+
+ {
+ ObjectLock olock(newConfig);
+
+ for (const Dictionary::Pair& kv : newConfig) {
+
+ /* Store the relative config file path for later validation and activation.
+ * IMPORTANT: Store this prior to any filters.
+ * */
+ relativePaths.push_back(zoneName + "/" + kv.first);
+
+ // Ignore same config content. This is an expensive comparison.
+ if (productionConfig->Get(kv.first) == kv.second)
+ continue;
+
+ String path = stageConfigZoneDir + "/" + kv.first;
+
+ if (Utility::Match("*.conf", path)) {
+ Log(LogInformation, "ApiListener")
+ << "Stage: Updating received configuration file '" << path << "' for zone '" << zoneName << "'.";
+ }
+
+ // Sync string content only.
+ String content = kv.second;
+
+ // Generate a directory tree (zones/1/2/3 might not exist yet).
+ Utility::MkDirP(Utility::DirName(path), 0755);
+
+ // Write the content to file.
+ std::ofstream fp(path.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
+ fp << content;
+ fp.close();
+
+ numBytes += content.GetLength();
+ }
+ }
+
+ Log(LogInformation, "ApiListener")
+ << "Applying configuration file update for path '" << stageConfigZoneDir << "' ("
+ << numBytes << " Bytes).";
+
+ // If the update removes a path, delete it on disk and signal a config change.
+ {
+ ObjectLock xlock(productionConfig);
+
+ for (const Dictionary::Pair& kv : productionConfig) {
+ if (!newConfig->Contains(kv.first)) {
+ configChange = true;
+
+ String path = stageConfigZoneDir + "/" + kv.first;
+ Utility::Remove(path);
+ }
+ }
+ }
}
+ /*
+ * We have processed all configuration files and stored them in the staging directory.
+ *
+ * We need to store them locally for later analysis. A config change means
+ * that we will validate the configuration in a separate process sandbox,
+ * and only copy the configuration to production when everything is ok.
+ *
+ * A successful validation also triggers the final restart.
+ */
if (configChange) {
- Log(LogInformation, "ApiListener", "Restarting after configuration change.");
- Application::RequestRestart();
+ Log(LogInformation, "ApiListener")
+ << "Received configuration from endpoint '" << fromEndpointName
+ << "' is different to production, triggering validation and reload.";
+ AsyncTryActivateZonesStage(relativePaths);
+ } else {
+ Log(LogInformation, "ApiListener")
+ << "Received configuration from endpoint '" << fromEndpointName
+ << "' is equal to production, not triggering reload.";
}
return Empty;
}
+
+/**
+ * Callback for stage config validation.
+ * When validation was successful, the configuration is copied from
+ * stage to production and a restart is triggered.
+ * On failure, there's no restart and this is logged.
+ *
+ * @param pr Result of the validation process.
+ * @param relativePaths Collected paths including the zone name, which are copied from stage to current directories.
+ */
+void ApiListener::TryActivateZonesStageCallback(const ProcessResult& pr,
+ const std::vector& relativePaths)
+{
+ String apiZonesDir = GetApiZonesDir();
+ String apiZonesStageDir = GetApiZonesStageDir();
+
+ String logFile = apiZonesStageDir + "/startup.log";
+ std::ofstream fpLog(logFile.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
+ fpLog << pr.Output;
+ fpLog.close();
+
+ String statusFile = apiZonesStageDir + "/status";
+ std::ofstream fpStatus(statusFile.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
+ fpStatus << pr.ExitStatus;
+ fpStatus.close();
+
+ // Validation went fine, copy stage and reload.
+ if (pr.ExitStatus == 0) {
+ Log(LogInformation, "ApiListener")
+ << "Config validation for stage '" << apiZonesStageDir << "' was OK, replacing into '" << apiZonesDir << "' and triggering reload.";
+
+ // Purge production before copying stage.
+ if (Utility::PathExists(apiZonesDir))
+ Utility::RemoveDirRecursive(apiZonesDir);
+
+ Utility::MkDirP(apiZonesDir, 0700);
+
+ // Copy all synced configuration files from stage to production.
+ for (const String& path : relativePaths) {
+ if (!Utility::PathExists(path))
+ continue;
+
+ Log(LogInformation, "ApiListener")
+ << "Copying file '" << path << "' from config sync staging to production zones directory.";
+
+ String stagePath = apiZonesStageDir + path;
+ String currentPath = apiZonesDir + path;
+
+ Utility::MkDirP(Utility::DirName(currentPath), 0700);
+
+ Utility::CopyFile(stagePath, currentPath);
+ }
+
+ // Clear any failed deployment before
+ ApiListener::Ptr listener = ApiListener::GetInstance();
+
+ if (listener)
+ listener->ClearLastFailedZonesStageValidation();
+
+ Application::RequestRestart();
+
+ // All good, return early.
+ return;
+ }
+
+ // Error case.
+ Log(LogCritical, "ApiListener")
+ << "Config validation failed for staged cluster config sync in '" << apiZonesStageDir
+ << "'. Aborting. Logs: '" << logFile << "'";
+
+ ApiListener::Ptr listener = ApiListener::GetInstance();
+
+ if (listener)
+ listener->UpdateLastFailedZonesStageValidation(pr.Output);
+}
+
+/**
+ * Spawns a new validation process and waits for its output.
+ * Sets 'System.ZonesStageVarDir' to override the config validation zone dirs with our current stage.
+ *
+ * @param relativePaths Required for later file operations in the callback. Provides the zone name plus path in a list.
+ */
+void ApiListener::AsyncTryActivateZonesStage(const std::vector& relativePaths)
+{
+ VERIFY(Application::GetArgC() >= 1);
+
+ /* Inherit parent process args. */
+ Array::Ptr args = new Array({
+ Application::GetExePath(Application::GetArgV()[0]),
+ });
+
+ for (int i = 1; i < Application::GetArgC(); i++) {
+ String argV = Application::GetArgV()[i];
+
+ if (argV == "-d" || argV == "--daemonize")
+ continue;
+
+ args->Add(argV);
+ }
+
+ args->Add("--validate");
+
+ // Set the ZonesStageDir. This creates our own local chroot without any additional automated zone includes.
+ args->Add("--define");
+ args->Add("System.ZonesStageVarDir=" + GetApiZonesStageDir());
+
+ Process::Ptr process = new Process(Process::PrepareCommand(args));
+ process->SetTimeout(Application::GetReloadTimeout());
+ process->Run(std::bind(&TryActivateZonesStageCallback, _1, relativePaths));
+}
+
+/**
+ * Update the structure from the last failed validation output.
+ * Uses the current timestamp.
+ *
+ * @param log The process output from the config validation.
+ */
+void ApiListener::UpdateLastFailedZonesStageValidation(const String& log)
+{
+ Dictionary::Ptr lastFailedZonesStageValidation = new Dictionary({
+ { "log", log },
+ { "ts", Utility::GetTime() }
+ });
+
+ SetLastFailedZonesStageValidation(lastFailedZonesStageValidation);
+}
+
+/**
+ * Clear the structure for the last failed reload.
+ *
+ */
+void ApiListener::ClearLastFailedZonesStageValidation()
+{
+ SetLastFailedZonesStageValidation(Dictionary::Ptr());
+}
+
+/**
+ * Generate a config checksum.
+ *
+ * @param content String content used for generating the checksum.
+ * @returns The checksum as string.
+ */
+String ApiListener::GetChecksum(const String& content)
+{
+ return SHA256(content);
+}
+
+bool ApiListener::CheckConfigChange(const ConfigDirInformation& oldConfig, const ConfigDirInformation& newConfig)
+{
+ Dictionary::Ptr oldChecksums = oldConfig.Checksums;
+ Dictionary::Ptr newChecksums = newConfig.Checksums;
+
+ // TODO: Figure out whether normal users need this for debugging.
+ Log(LogDebug, "ApiListener")
+ << "Checking for config change between stage and production. Old (" << oldChecksums->GetLength() << "): '"
+ << JsonEncode(oldChecksums)
+ << "' vs. new (" << newChecksums->GetLength() << "): '"
+ << JsonEncode(newChecksums) << "'.";
+
+ // Different length means that either one or the other side added or removed something. */
+ if (oldChecksums->GetLength() != newChecksums->GetLength())
+ return true;
+
+ // Both dictionaries have an equal size.
+ ObjectLock olock(oldChecksums);
+
+ for (const Dictionary::Pair& kv : oldChecksums) {
+ String path = kv.first;
+ String oldChecksum = kv.second;
+
+ // TODO: Figure out if config changes only apply to '.conf'. Leaving this open for other config files.
+ //if (!Utility::Match("*.conf", path))
+ // continue;
+
+ /* Ignore internal files, especially .timestamp and .checksums.
+ *
+ * If we don't, this results in "always change" restart loops.
+ */
+ if (Utility::Match("/.*", path))
+ continue;
+
+ Log(LogDebug, "ApiListener")
+ << "Checking " << path << " for checksum: " << oldChecksum;
+
+ // Check whether our key exists in the new checksums, and they have an equal value.
+ String newChecksum = newChecksums->Get(path);
+
+ if (newChecksums->Get(path) != kv.second) {
+ Log(LogDebug, "ApiListener")
+ << "Path '" << path << "' doesn't match old checksum '"
+ << newChecksum << "' with new checksum '" << oldChecksum << "'.";
+ return true;
+ }
+ }
+
+ return false;
+}
+
+/**
+ * Load the given config dir and read their file content into the config structure.
+ *
+ * @param dir Path to the config directory.
+ * @returns ConfigDirInformation structure.
+ */
+ConfigDirInformation ApiListener::LoadConfigDir(const String& dir)
+{
+ ConfigDirInformation config;
+ config.UpdateV1 = new Dictionary();
+ config.UpdateV2 = new Dictionary();
+ config.Checksums = new Dictionary();
+
+ Utility::GlobRecursive(dir, "*", std::bind(&ApiListener::ConfigGlobHandler, std::ref(config), dir, _1), GlobFile);
+ return config;
+}
+
+/**
+ * Read the given file and store it in the config information structure.
+ * Callback function for Glob().
+ *
+ * @param config Reference to the config information object.
+ * @param path File path.
+ * @param file Full file name.
+ */
+void ApiListener::ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file)
+{
+ // Avoid loading the authoritative marker for syncs at all cost.
+ if (Utility::BaseName(file) == ".authoritative")
+ return;
+
+ CONTEXT("Creating config update for file '" + file + "'");
+
+ Log(LogNotice, "ApiListener")
+ << "Creating config update for file '" << file << "'.";
+
+ std::ifstream fp(file.CStr(), std::ifstream::binary);
+ if (!fp)
+ return;
+
+ String content((std::istreambuf_iterator(fp)), std::istreambuf_iterator());
+
+ Dictionary::Ptr update;
+ String relativePath = file.SubStr(path.GetLength());
+
+ /*
+ * 'update' messages contain conf files. 'update_v2' syncs everything else (.timestamp).
+ *
+ * **Keep this intact to stay compatible with older clients.**
+ */
+ if (Utility::Match("*.conf", file))
+ update = config.UpdateV1;
+ else
+ update = config.UpdateV2;
+
+ update->Set(relativePath, content);
+
+ /* Calculate a checksum for each file (and a global one later).
+ *
+ * IMPORTANT: Ignore the .authoritative file above, this must not be synced.
+ * */
+ config.Checksums->Set(relativePath, GetChecksum(content));
+}
+
+/**
+ * Compatibility helper for merging config update v1 and v2 into a global result.
+ *
+ * @param config Config information structure.
+ * @returns Dictionary which holds the merged information.
+ */
+Dictionary::Ptr ApiListener::MergeConfigUpdate(const ConfigDirInformation& config)
+{
+ Dictionary::Ptr result = new Dictionary();
+
+ if (config.UpdateV1)
+ config.UpdateV1->CopyTo(result);
+
+ if (config.UpdateV2)
+ config.UpdateV2->CopyTo(result);
+
+ return result;
+}
diff --git a/lib/remote/apilistener.cpp b/lib/remote/apilistener.cpp
index a7f0f66a7..661a4eaaf 100644
--- a/lib/remote/apilistener.cpp
+++ b/lib/remote/apilistener.cpp
@@ -59,6 +59,16 @@ String ApiListener::GetApiDir()
return Configuration::DataDir + "/api/";
}
+String ApiListener::GetApiZonesDir()
+{
+ return GetApiDir() + "zones/";
+}
+
+String ApiListener::GetApiZonesStageDir()
+{
+ return GetApiDir() + "zones-stage/";
+}
+
String ApiListener::GetCertsDir()
{
return Configuration::DataDir + "/certs/";
@@ -232,7 +242,7 @@ void ApiListener::Start(bool runtimeCreated)
Log(LogInformation, "ApiListener")
<< "'" << GetName() << "' started.";
- SyncZoneDirs();
+ SyncLocalZoneDirs();
ObjectImpl::Start(runtimeCreated);
diff --git a/lib/remote/apilistener.hpp b/lib/remote/apilistener.hpp
index 022ad8281..0f552e984 100644
--- a/lib/remote/apilistener.hpp
+++ b/lib/remote/apilistener.hpp
@@ -9,6 +9,7 @@
#include "remote/endpoint.hpp"
#include "remote/messageorigin.hpp"
#include "base/configobject.hpp"
+#include "base/process.hpp"
#include "base/timer.hpp"
#include "base/workqueue.hpp"
#include "base/tcpsocket.hpp"
@@ -31,6 +32,7 @@ struct ConfigDirInformation
{
Dictionary::Ptr UpdateV1;
Dictionary::Ptr UpdateV2;
+ Dictionary::Ptr Checksums;
};
/**
@@ -47,6 +49,8 @@ public:
ApiListener();
static String GetApiDir();
+ static String GetApiZonesDir();
+ static String GetApiZonesStageDir();
static String GetCertsDir();
static String GetCaDir();
static String GetCertificateRequestsDir();
@@ -167,16 +171,26 @@ private:
/* filesync */
static boost::mutex m_ConfigSyncStageLock;
- static ConfigDirInformation LoadConfigDir(const String& dir);
- static Dictionary::Ptr MergeConfigUpdate(const ConfigDirInformation& config);
- static bool UpdateConfigDir(const ConfigDirInformation& oldConfig, const ConfigDirInformation& newConfig, const String& configDir, bool authoritative);
+ void SyncLocalZoneDirs() const;
+ void SyncLocalZoneDir(const Zone::Ptr& zone) const;
- void SyncZoneDirs() const;
- void SyncZoneDir(const Zone::Ptr& zone) const;
-
- static void ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file);
void SendConfigUpdate(const JsonRpcConnection::Ptr& aclient);
+ static Dictionary::Ptr MergeConfigUpdate(const ConfigDirInformation& config);
+
+ static ConfigDirInformation LoadConfigDir(const String& dir);
+ static void ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file);
+
+ static void TryActivateZonesStageCallback(const ProcessResult& pr,
+ const std::vector& relativePaths);
+ static void AsyncTryActivateZonesStage(const std::vector& relativePaths);
+
+ static String GetChecksum(const String& content);
+ static bool CheckConfigChange(const ConfigDirInformation& oldConfig, const ConfigDirInformation& newConfig);
+
+ void UpdateLastFailedZonesStageValidation(const String& log);
+ void ClearLastFailedZonesStageValidation();
+
/* configsync */
void UpdateConfigObject(const ConfigObject::Ptr& object, const MessageOrigin::Ptr& origin,
const JsonRpcConnection::Ptr& client = nullptr);
diff --git a/lib/remote/apilistener.ti b/lib/remote/apilistener.ti
index 4217ce0ab..ac17ccfcf 100644
--- a/lib/remote/apilistener.ti
+++ b/lib/remote/apilistener.ti
@@ -54,6 +54,8 @@ class ApiListener : ConfigObject
[state, no_user_modify] Timestamp log_message_timestamp;
[no_user_modify] String identity;
+
+ [state, no_user_modify] Dictionary::Ptr last_failed_zones_stage_validation;
};
}