mirror of https://github.com/Icinga/icinga2.git
Merge pull request #6727 from Icinga/feature/cluster-config-sync-stage
Improve cluster config sync
This commit is contained in:
Michael Friedrich
GitHub
commit
0b85928a30
|
|
@ -96,7 +96,33 @@ feature with the `cipher_list` attribute.
|
|||
In case that one of these ciphers is marked as insecure in the future,
|
||||
please let us know with an issue on GitHub.
|
||||
|
||||
### HA-aware Features <a id="upgrading-to-2-11-ha-aware-features"></a>
|
||||
### Cluster <a id="upgrading-to-2-11-cluster"></a>
|
||||
|
||||
#### Config Sync <a id="upgrading-to-2-11-cluster-config-sync"></a>
|
||||
|
||||
2.11 overhauls the cluster config sync in many ways. This includes the
|
||||
following under the hood:
|
||||
|
||||
- Synced configuration files are not immediately put into production, but left inside a stage.
|
||||
- Unsuccessful config validation never puts the config into production, additional logging and API states are available.
|
||||
- Zone directories which are not configured in zones.conf, are not included anymore on secondary master/satellites/clients.
|
||||
- Synced config change calculation use checksums instead of timestamps to trigger validation/reload. This is more safe, and the usage of timestamps is now deprecated.
|
||||
- Don't allow parallel cluster syncs to avoid race conditions with overridden files.
|
||||
- Deleted directories and files are now purged, previous versions had a bug.
|
||||
|
||||
Whenever a newer child endpoint receives a configuration update without
|
||||
checksums, it will log a warning.
|
||||
|
||||
```
|
||||
Received configuration update without checksums from parent endpoint satellite1. This behaviour is deprecated. Please upgrade the parent endpoint to 2.11+
|
||||
```
|
||||
|
||||
This is a gentle reminder to upgrade the master and satellites first,
|
||||
prior to installing new clients/agents.
|
||||
|
||||
Technical details are available in the [technical concepts](19-technical-concepts.md#technical-concepts-cluster-config-sync) chapter.
|
||||
|
||||
#### HA-aware Features <a id="upgrading-to-2-11-cluster-ha-aware-features"></a>
|
||||
|
||||
v2.11 introduces additional HA functionality similar to the DB IDO feature.
|
||||
This enables the feature being active only on one endpoint while the other
|
||||
|
|
@ -182,7 +208,7 @@ constant in [constants.conf](04-configuring-icinga-2.md#constants-conf) instead.
|
|||
|
||||
### REST API <a id="upgrading-to-2-11-api"></a>
|
||||
|
||||
#### Actions <a id="upgrading-to-2-11-api-config-packages"></a>
|
||||
#### Actions <a id="upgrading-to-2-11-api-actions"></a>
|
||||
|
||||
The [schedule-downtime](12-icinga2-api.md#icinga2-api-actions-schedule-downtime-host-all-services)
|
||||
action supports the `all_services` parameter for Host types. Defaults to false.
|
||||
|
|
|
|||
|
|
@ -813,6 +813,188 @@ Icinga 2 v2.9+ adds more performance metrics for these values:
|
|||
* `sum_bytes_sent_per_second` and `sum_bytes_received_per_second`
|
||||
|
||||
|
||||
### Config Sync <a id="technical-concepts-cluster-config-sync"></a>
|
||||
|
||||
The visible feature for the user is to put configuration files in `/etc/icinga2/zones.d/<zonename>`
|
||||
and have them synced automatically to all involved zones and endpoints.
|
||||
|
||||
This not only includes host and service objects being checked
|
||||
in a satellite zone, but also additional config objects such as
|
||||
commands, groups, timeperiods and also templates.
|
||||
|
||||
Additional thoughts and complexity added:
|
||||
|
||||
- Putting files into zone directory names removes the burden to set the `zone` attribute on each object in this directory. This is done automatically by the config compiler.
|
||||
- Inclusion of `zones.d` happens automatically, the user shouldn't be bothered about this.
|
||||
- Before the REST API was created, only static configuration files in `/etc/icinga2/zones.d` existed. With the addition of config packages, additional `zones.d` targets must be registered (e.g. used by the Director)
|
||||
- Only one config master is allowed. This one identifies itself with configuration files in `/etc/icinga2/zones.d`. This is not necessarily the zone master seen in the debug logs, that one is important for message routing internally.
|
||||
- Objects and templates which cannot be bound into a specific zone (e.g. hosts in the satellite zone) must be made available "globally".
|
||||
- Users must be able to deny the synchronisation of specific zones, e.g. for security reasons.
|
||||
|
||||
#### Config Sync: Config Master <a id="technical-concepts-cluster-config-sync-config-master"></a>
|
||||
|
||||
All zones must be configured and included in the `zones.conf` config file beforehand.
|
||||
The zone names are the identifier for the directories underneath the `/etc/icinga2/zones.d`
|
||||
directory. If a zone is not configured, it will not be included in the config sync - keep this
|
||||
in mind for troubleshooting.
|
||||
|
||||
When the config master starts, the content of `/etc/icinga2/zones.d` is automatically
|
||||
included. There's no need for an additional entry in `icinga2.conf` like `conf.d`.
|
||||
You can verify this by running the config validation on debug level:
|
||||
|
||||
```
|
||||
icinga2 daemon -C -x debug | grep 'zones.d'
|
||||
|
||||
[2019-06-19 15:16:19 +0200] notice/ConfigCompiler: Compiling config file: /etc/icinga2/zones.d/global-templates/commands.conf
|
||||
```
|
||||
|
||||
Once the config validation succeeds, the startup routine for the daemon
|
||||
copies the files into the "production" directory in `/var/lib/icinga2/api/zones`.
|
||||
This directory is used for all endpoints where Icinga stores the received configuration.
|
||||
With the exception of the config master retrieving this from `/etc/icinga2/zones.d` instead.
|
||||
|
||||
These operations are logged for better visibility.
|
||||
|
||||
```
|
||||
[2019-06-19 15:26:38 +0200] information/ApiListener: Copying 1 zone configuration files for zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'.
|
||||
[2019-06-19 15:26:38 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/commands.conf
|
||||
```
|
||||
|
||||
The master is finished at this point. Depending on the cluster configuration,
|
||||
the next iteration is a connected endpoint after successful TLS handshake and certificate
|
||||
authentication.
|
||||
|
||||
It calls `SendConfigUpdate(client)` which sends the [config::Update](19-technical-concepts.md#technical-concepts-json-rpc-messages-config-update)
|
||||
JSON-RPC message including all required zones and their configuration file content.
|
||||
|
||||
|
||||
#### Config Sync: Receive Config <a id="technical-concepts-cluster-config-sync-receive-config"></a>
|
||||
|
||||
The secondary master endpoint and endpoints in a child zone will be connected to the config
|
||||
master. The endpoint receives the [config::Update](19-technical-concepts.md#technical-concepts-json-rpc-messages-config-update)
|
||||
JSON-RPC message and processes the content in `ConfigUpdateHandler()`. This method checks
|
||||
whether config should be accepted. In addition to that, it locks a local mutex to avoid race conditions
|
||||
with multiple syncs in parallel.
|
||||
|
||||
After that, the received configuration content is analysed.
|
||||
|
||||
> **Note**
|
||||
>
|
||||
> The cluster design allows that satellite endpoints may connect to the secondary master first.
|
||||
> There is no immediate need to always connect to the config master first, especially since
|
||||
> the satellite endpoints don't know that.
|
||||
>
|
||||
> The secondary master not only stores the master zone config files, but also all child zones.
|
||||
> This is also the case for any HA enabled zone with more than one endpoint.
|
||||
|
||||
|
||||
2.11 puts the received configuration files into a staging directory in
|
||||
`/var/lib/icinga2/api/zones-stage`. Previous versions directly wrote the
|
||||
files into production which could have led to broken configuration on the
|
||||
next manual restart.
|
||||
|
||||
```
|
||||
[2019-06-19 16:08:29 +0200] information/ApiListener: New client connection for identity 'master1' to [127.0.0.1]:5665
|
||||
[2019-06-19 16:08:30 +0200] information/ApiListener: Applying config update from endpoint 'master1' of zone 'master'.
|
||||
[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master1'. Comparing the checksums.
|
||||
[2019-06-19 16:08:30 +0200] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/agent//_etc/host.conf' for zone 'agent'.
|
||||
[2019-06-19 16:08:30 +0200] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/agent' (176 Bytes).
|
||||
[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration for zone 'master' from endpoint 'master1'. Comparing the checksums.
|
||||
[2019-06-19 16:08:30 +0200] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/master' (17 Bytes).
|
||||
[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration from endpoint 'master1' is different to production, triggering validation and reload.
|
||||
```
|
||||
|
||||
It then validates the received configuration in its own config stage. There is
|
||||
an parameter override in place which disables the automatic inclusion of the production
|
||||
config in `/var/lib/icinga2/api/zones`.
|
||||
|
||||
Once completed, the reload is triggered. This follows the same configurable timeout
|
||||
as with the global reload.
|
||||
|
||||
```
|
||||
[2019-06-19 16:52:26 +0200] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
|
||||
[2019-06-19 16:52:27 +0200] information/Application: Got reload command: Started new instance with PID '19945' (timeout is 300s).
|
||||
[2019-06-19 16:52:28 +0200] information/Application: Reload requested, letting new process take over.
|
||||
```
|
||||
|
||||
Whenever the staged configuration validation fails, Icinga logs this including a reference
|
||||
to the startup log file which includes additional errors.
|
||||
|
||||
```
|
||||
[2019-06-19 15:45:27 +0200] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Aborting. Logs: '/var/lib/icinga2/api/zones-stage//startup.log'
|
||||
```
|
||||
|
||||
|
||||
#### Config Sync: Changes and Reload <a id="technical-concepts-cluster-config-sync-changes-reload"></a>
|
||||
|
||||
Whenever a new configuration is received, it is validated and upon success, the
|
||||
daemon automatically reloads. While the daemon continues with checks, the reload
|
||||
cannot hand over open TCP connections. That being said, reloading the daemon everytime
|
||||
a configuration is synchronized would lead into many not connected endpoints.
|
||||
|
||||
Therefore the cluster config sync checks whether the configuration files actually
|
||||
changed, and will only trigger a reload when such a change happened.
|
||||
|
||||
2.11 calculates a checksum from each file content and compares this to the
|
||||
production configuration. Previous versions used additional metadata with timestamps from
|
||||
files which sometimes led to problems with asynchronous dates.
|
||||
|
||||
> **Note**
|
||||
>
|
||||
> For compatibility reasons, the timestamp metadata algorithm is still intact, e.g.
|
||||
> when the client is 2.11 already, but the parent endpoint is still on 2.10.
|
||||
|
||||
Icinga logs a warning when this happens.
|
||||
|
||||
```
|
||||
Received configuration update without checksums from parent endpoint satellite1. This behaviour is deprecated. Please upgrade the parent endpoint to 2.11+
|
||||
```
|
||||
|
||||
|
||||
The debug log provides more details on the actual checksums and checks. Future output
|
||||
may change, use this solely for troubleshooting and debugging whenever the cluster
|
||||
config sync fails.
|
||||
|
||||
```
|
||||
[2019-06-19 16:13:16 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master1'. Comparing the checksums.
|
||||
[2019-06-19 16:13:16 +0200] debug/ApiListener: Checking for config change between stage and production. Old (3): '{"/.checksums":"7ede1276a9a32019c1412a52779804a976e163943e268ec4066e6b6ec4d15d73","/.timestamp":"ec4354b0eca455f7c2ca386fddf5b9ea810d826d402b3b6ac56ba63b55c2892c","/_etc/host.conf":"35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44"}' vs. new (3): '{"/.checksums":"84a586435d732327e2152e7c9b6d85a340cc917b89ae30972042f3dc344ea7cf","/.timestamp":"0fd6facf35e49ab1b2a161872fa7ad794564eba08624373d99d31c32a7a4c7d3","/_etc/host.conf":"0d62075e89be14088de1979644b40f33a8f185fcb4bb6ff1f7da2f63c7723fcb"}'.
|
||||
[2019-06-19 16:13:16 +0200] debug/ApiListener: Checking /_etc/host.conf for checksum: 35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44
|
||||
[2019-06-19 16:13:16 +0200] debug/ApiListener: Path '/_etc/host.conf' doesn't match old checksum '0d62075e89be14088de1979644b40f33a8f185fcb4bb6ff1f7da2f63c7723fcb' with new checksum '35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44'.
|
||||
```
|
||||
|
||||
|
||||
#### Config Sync: Trust <a id="technical-concepts-cluster-config-sync-trust"></a>
|
||||
|
||||
The config sync follows the "top down" approach, where the master endpoint in the master
|
||||
zone is allowed to synchronize configuration to the child zone, e.g. the satellite zone.
|
||||
|
||||
Endpoints in the same zone, e.g. a secondary master, receive configuration for the same
|
||||
zone and all child zones.
|
||||
|
||||
Endpoints in the satellite zone trust the parent zone, and will accept the pushed
|
||||
configuration via JSON-RPC cluster messages. By default, this is disabled and must
|
||||
be enabled with the `accept_config` attribute in the ApiListener feature (manually or with CLI
|
||||
helpers).
|
||||
|
||||
The satellite zone will not only accept zone configuration for its own zone, but also
|
||||
all configured child zones. That is why it is important to configure the zone hierarchy
|
||||
on the satellite as well.
|
||||
|
||||
Child zones are not allowed to sync configuration up to the parent zone. Each Icinga instance
|
||||
evaluates this in startup and knows on endpoint connect which config zones need to be synced.
|
||||
|
||||
|
||||
Global zones have a special trust relationship: They are synced to all child zones, be it
|
||||
a satellite zone or client zone. Since checkable objects such as a Host or a Service object
|
||||
must have only one endpoint as authority, they cannot be put into a global zone (denied by
|
||||
the config compiler).
|
||||
|
||||
Apply rules and templates are allowed, since they are evaluated in the endpoint which received
|
||||
the synced configuration. Keep in mind that there may be differences on the master and the satellite
|
||||
when e.g. hostgroup membership is used for assign where expressions, but the groups are only
|
||||
available on the master.
|
||||
|
||||
|
||||
## TLS Network IO <a id="technical-concepts-tls-network-io"></a>
|
||||
|
||||
### TLS Connection Handling <a id="technical-concepts-tls-network-io-connection-handling"></a>
|
||||
|
|
|
|||
|
|
@ -100,6 +100,10 @@ static void IncludePackage(const String& packagePath, bool& success)
|
|||
bool DaemonUtility::ValidateConfigFiles(const std::vector<std::string>& configs, const String& objectsFile)
|
||||
{
|
||||
bool success;
|
||||
|
||||
Namespace::Ptr systemNS = ScriptGlobal::Get("System");
|
||||
VERIFY(systemNS);
|
||||
|
||||
if (!objectsFile.IsEmpty())
|
||||
ConfigCompilerContext::GetInstance()->OpenObjectsFile(objectsFile);
|
||||
|
||||
|
|
@ -122,12 +126,15 @@ bool DaemonUtility::ValidateConfigFiles(const std::vector<std::string>& configs,
|
|||
* unfortunately moving it there is somewhat non-trivial. */
|
||||
success = true;
|
||||
|
||||
String zonesEtcDir = Configuration::ZonesDir;
|
||||
if (!zonesEtcDir.IsEmpty() && Utility::PathExists(zonesEtcDir))
|
||||
Utility::Glob(zonesEtcDir + "/*", std::bind(&IncludeZoneDirRecursive, _1, "_etc", std::ref(success)), GlobDirectory);
|
||||
/* Only load zone directory if we're not in staging validation. */
|
||||
if (!systemNS->Contains("ZonesStageVarDir")) {
|
||||
String zonesEtcDir = Configuration::ZonesDir;
|
||||
if (!zonesEtcDir.IsEmpty() && Utility::PathExists(zonesEtcDir))
|
||||
Utility::Glob(zonesEtcDir + "/*", std::bind(&IncludeZoneDirRecursive, _1, "_etc", std::ref(success)), GlobDirectory);
|
||||
|
||||
if (!success)
|
||||
return false;
|
||||
if (!success)
|
||||
return false;
|
||||
}
|
||||
|
||||
/* Load package config files - they may contain additional zones which
|
||||
* are authoritative on this node and are checked in HasZoneConfigAuthority(). */
|
||||
|
|
@ -138,17 +145,24 @@ bool DaemonUtility::ValidateConfigFiles(const std::vector<std::string>& configs,
|
|||
if (!success)
|
||||
return false;
|
||||
|
||||
/* Load cluster synchronized configuration files */
|
||||
/* Load cluster synchronized configuration files. This can be overridden for staged sync validations. */
|
||||
String zonesVarDir = Configuration::DataDir + "/api/zones";
|
||||
|
||||
/* Cluster config sync stage validation needs this. */
|
||||
if (systemNS->Contains("ZonesStageVarDir")) {
|
||||
zonesVarDir = systemNS->Get("ZonesStageVarDir");
|
||||
|
||||
Log(LogNotice, "DaemonUtility")
|
||||
<< "Overriding zones var directory with '" << zonesVarDir << "' for cluster config sync staging.";
|
||||
}
|
||||
|
||||
|
||||
if (Utility::PathExists(zonesVarDir))
|
||||
Utility::Glob(zonesVarDir + "/*", std::bind(&IncludeNonLocalZone, _1, "_cluster", std::ref(success)), GlobDirectory);
|
||||
|
||||
if (!success)
|
||||
return false;
|
||||
|
||||
Namespace::Ptr systemNS = ScriptGlobal::Get("System");
|
||||
VERIFY(systemNS);
|
||||
|
||||
/* This is initialized inside the IcingaApplication class. */
|
||||
Value vAppType;
|
||||
VERIFY(systemNS->Get("ApplicationType", &vAppType));
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@
|
|||
#include "icinga/icingaapplication.hpp"
|
||||
#include "icinga/clusterevents.hpp"
|
||||
#include "icinga/checkable.hpp"
|
||||
#include "remote/apilistener.hpp"
|
||||
#include "base/application.hpp"
|
||||
#include "base/objectlock.hpp"
|
||||
#include "base/utility.hpp"
|
||||
|
|
@ -157,6 +158,20 @@ void IcingaCheckTask::ScriptFunc(const Checkable::Ptr& checkable, const CheckRes
|
|||
cr->SetState(ServiceWarning);
|
||||
}
|
||||
|
||||
/* Indicate a warning when the last synced config caused a stage validation error. */
|
||||
ApiListener::Ptr listener = ApiListener::GetInstance();
|
||||
|
||||
if (listener) {
|
||||
Dictionary::Ptr validationResult = listener->GetLastFailedZonesStageValidation();
|
||||
|
||||
if (validationResult) {
|
||||
output += "; Last zone sync stage validation failed at "
|
||||
+ Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", validationResult->Get("ts"));
|
||||
|
||||
cr->SetState(ServiceWarning);
|
||||
}
|
||||
}
|
||||
|
||||
/* Extract the version number of the running Icinga2 instance.
|
||||
* We assume that appVersion will allways be something like 'v2.10.1-8-gaebe6da' and we want to extract '2.10.1'.
|
||||
*/
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@ set_target_properties (
|
|||
#install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/api\")")
|
||||
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/api/log\")")
|
||||
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/api/zones\")")
|
||||
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/api/zones-stage\")")
|
||||
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/certs\")")
|
||||
install(CODE "file(MAKE_DIRECTORY \"\$ENV{DESTDIR}${ICINGA2_FULL_DATADIR}/certificate-requests\")")
|
||||
|
||||
|
|
|
|||
|
|
@ -3,9 +3,12 @@
|
|||
#include "remote/apilistener.hpp"
|
||||
#include "remote/apifunction.hpp"
|
||||
#include "config/configcompiler.hpp"
|
||||
#include "base/tlsutility.hpp"
|
||||
#include "base/json.hpp"
|
||||
#include "base/configtype.hpp"
|
||||
#include "base/logger.hpp"
|
||||
#include "base/convert.hpp"
|
||||
#include "base/application.hpp"
|
||||
#include "base/exception.hpp"
|
||||
#include "base/utility.hpp"
|
||||
#include <fstream>
|
||||
|
|
@ -17,233 +20,200 @@ REGISTER_APIFUNCTION(Update, config, &ApiListener::ConfigUpdateHandler);
|
|||
|
||||
boost::mutex ApiListener::m_ConfigSyncStageLock;
|
||||
|
||||
void ApiListener::ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file)
|
||||
{
|
||||
CONTEXT("Creating config update for file '" + file + "'");
|
||||
|
||||
Log(LogNotice, "ApiListener")
|
||||
<< "Creating config update for file '" << file << "'.";
|
||||
|
||||
std::ifstream fp(file.CStr(), std::ifstream::binary);
|
||||
if (!fp)
|
||||
return;
|
||||
|
||||
String content((std::istreambuf_iterator<char>(fp)), std::istreambuf_iterator<char>());
|
||||
|
||||
Dictionary::Ptr update;
|
||||
|
||||
if (Utility::Match("*.conf", file))
|
||||
update = config.UpdateV1;
|
||||
else
|
||||
update = config.UpdateV2;
|
||||
|
||||
update->Set(file.SubStr(path.GetLength()), content);
|
||||
}
|
||||
|
||||
Dictionary::Ptr ApiListener::MergeConfigUpdate(const ConfigDirInformation& config)
|
||||
{
|
||||
Dictionary::Ptr result = new Dictionary();
|
||||
|
||||
if (config.UpdateV1)
|
||||
config.UpdateV1->CopyTo(result);
|
||||
|
||||
if (config.UpdateV2)
|
||||
config.UpdateV2->CopyTo(result);
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
ConfigDirInformation ApiListener::LoadConfigDir(const String& dir)
|
||||
{
|
||||
ConfigDirInformation config;
|
||||
config.UpdateV1 = new Dictionary();
|
||||
config.UpdateV2 = new Dictionary();
|
||||
Utility::GlobRecursive(dir, "*", std::bind(&ApiListener::ConfigGlobHandler, std::ref(config), dir, _1), GlobFile);
|
||||
return config;
|
||||
}
|
||||
|
||||
bool ApiListener::UpdateConfigDir(const ConfigDirInformation& oldConfigInfo, const ConfigDirInformation& newConfigInfo, const String& configDir, bool authoritative)
|
||||
{
|
||||
bool configChange = false;
|
||||
|
||||
Dictionary::Ptr oldConfig = MergeConfigUpdate(oldConfigInfo);
|
||||
Dictionary::Ptr newConfig = MergeConfigUpdate(newConfigInfo);
|
||||
|
||||
double oldTimestamp;
|
||||
|
||||
if (!oldConfig->Contains("/.timestamp"))
|
||||
oldTimestamp = 0;
|
||||
else
|
||||
oldTimestamp = oldConfig->Get("/.timestamp");
|
||||
|
||||
double newTimestamp;
|
||||
|
||||
if (!newConfig->Contains("/.timestamp"))
|
||||
newTimestamp = Utility::GetTime();
|
||||
else
|
||||
newTimestamp = newConfig->Get("/.timestamp");
|
||||
|
||||
/* skip update if our configuration files are more recent */
|
||||
if (oldTimestamp >= newTimestamp) {
|
||||
Log(LogNotice, "ApiListener")
|
||||
<< "Our configuration is more recent than the received configuration update."
|
||||
<< " Ignoring configuration file update for path '" << configDir << "'. Current timestamp '"
|
||||
<< Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", oldTimestamp) << "' ("
|
||||
<< std::fixed << std::setprecision(6) << oldTimestamp
|
||||
<< ") >= received timestamp '"
|
||||
<< Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", newTimestamp) << "' ("
|
||||
<< newTimestamp << ").";
|
||||
return false;
|
||||
}
|
||||
|
||||
size_t numBytes = 0;
|
||||
|
||||
{
|
||||
ObjectLock olock(newConfig);
|
||||
for (const Dictionary::Pair& kv : newConfig) {
|
||||
if (oldConfig->Get(kv.first) != kv.second) {
|
||||
if (!Utility::Match("*/.timestamp", kv.first))
|
||||
configChange = true;
|
||||
|
||||
String path = configDir + "/" + kv.first;
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Updating configuration file: " << path;
|
||||
|
||||
/* Sync string content only. */
|
||||
String content = kv.second;
|
||||
|
||||
/* Generate a directory tree (zones/1/2/3 might not exist yet). */
|
||||
Utility::MkDirP(Utility::DirName(path), 0755);
|
||||
std::ofstream fp(path.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
|
||||
fp << content;
|
||||
fp.close();
|
||||
|
||||
numBytes += content.GetLength();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Applying configuration file update for path '" << configDir << "' (" << numBytes << " Bytes). Received timestamp '"
|
||||
<< Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", newTimestamp) << "' ("
|
||||
<< std::fixed << std::setprecision(6) << newTimestamp
|
||||
<< "), Current timestamp '"
|
||||
<< Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", oldTimestamp) << "' ("
|
||||
<< oldTimestamp << ").";
|
||||
|
||||
ObjectLock xlock(oldConfig);
|
||||
for (const Dictionary::Pair& kv : oldConfig) {
|
||||
if (!newConfig->Contains(kv.first)) {
|
||||
configChange = true;
|
||||
|
||||
String path = configDir + "/" + kv.first;
|
||||
(void) unlink(path.CStr());
|
||||
}
|
||||
}
|
||||
|
||||
String tsPath = configDir + "/.timestamp";
|
||||
if (!Utility::PathExists(tsPath)) {
|
||||
std::ofstream fp(tsPath.CStr(), std::ofstream::out | std::ostream::trunc);
|
||||
fp << std::fixed << newTimestamp;
|
||||
fp.close();
|
||||
}
|
||||
|
||||
if (authoritative) {
|
||||
String authPath = configDir + "/.authoritative";
|
||||
if (!Utility::PathExists(authPath)) {
|
||||
std::ofstream fp(authPath.CStr(), std::ofstream::out | std::ostream::trunc);
|
||||
fp.close();
|
||||
}
|
||||
}
|
||||
|
||||
return configChange;
|
||||
}
|
||||
|
||||
void ApiListener::SyncZoneDir(const Zone::Ptr& zone) const
|
||||
{
|
||||
ConfigDirInformation newConfigInfo;
|
||||
newConfigInfo.UpdateV1 = new Dictionary();
|
||||
newConfigInfo.UpdateV2 = new Dictionary();
|
||||
|
||||
for (const ZoneFragment& zf : ConfigCompiler::GetZoneDirs(zone->GetName())) {
|
||||
ConfigDirInformation newConfigPart = LoadConfigDir(zf.Path);
|
||||
|
||||
{
|
||||
ObjectLock olock(newConfigPart.UpdateV1);
|
||||
for (const Dictionary::Pair& kv : newConfigPart.UpdateV1) {
|
||||
newConfigInfo.UpdateV1->Set("/" + zf.Tag + kv.first, kv.second);
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
ObjectLock olock(newConfigPart.UpdateV2);
|
||||
for (const Dictionary::Pair& kv : newConfigPart.UpdateV2) {
|
||||
newConfigInfo.UpdateV2->Set("/" + zf.Tag + kv.first, kv.second);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
int sumUpdates = newConfigInfo.UpdateV1->GetLength() + newConfigInfo.UpdateV2->GetLength();
|
||||
|
||||
if (sumUpdates == 0)
|
||||
return;
|
||||
|
||||
String oldDir = Configuration::DataDir + "/api/zones/" + zone->GetName();
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Copying " << sumUpdates << " zone configuration files for zone '" << zone->GetName() << "' to '" << oldDir << "'.";
|
||||
|
||||
Utility::MkDirP(oldDir, 0700);
|
||||
|
||||
ConfigDirInformation oldConfigInfo = LoadConfigDir(oldDir);
|
||||
|
||||
UpdateConfigDir(oldConfigInfo, newConfigInfo, oldDir, true);
|
||||
}
|
||||
|
||||
void ApiListener::SyncZoneDirs() const
|
||||
/**
|
||||
* Entrypoint for updating all authoritative configs from /etc/zones.d, packages, etc.
|
||||
* into var/lib/icinga2/api/zones
|
||||
*/
|
||||
void ApiListener::SyncLocalZoneDirs() const
|
||||
{
|
||||
for (const Zone::Ptr& zone : ConfigType::GetObjectsByType<Zone>()) {
|
||||
try {
|
||||
SyncZoneDir(zone);
|
||||
SyncLocalZoneDir(zone);
|
||||
} catch (const std::exception&) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sync a zone directory where we have an authoritative copy (zones.d, packages, etc.)
|
||||
*
|
||||
* This function collects the registered zone config dirs from
|
||||
* the config compiler and reads the file content into the config
|
||||
* information structure.
|
||||
*
|
||||
* Returns early when there are no updates.
|
||||
*
|
||||
* @param zone Pointer to the zone object being synced.
|
||||
*/
|
||||
void ApiListener::SyncLocalZoneDir(const Zone::Ptr& zone) const
|
||||
{
|
||||
if (!zone)
|
||||
return;
|
||||
|
||||
ConfigDirInformation newConfigInfo;
|
||||
newConfigInfo.UpdateV1 = new Dictionary();
|
||||
newConfigInfo.UpdateV2 = new Dictionary();
|
||||
newConfigInfo.Checksums = new Dictionary();
|
||||
|
||||
String zoneName = zone->GetName();
|
||||
|
||||
// Load registered zone paths, e.g. '_etc', '_api' and user packages.
|
||||
for (const ZoneFragment& zf : ConfigCompiler::GetZoneDirs(zoneName)) {
|
||||
ConfigDirInformation newConfigPart = LoadConfigDir(zf.Path);
|
||||
|
||||
// Config files '*.conf'.
|
||||
{
|
||||
ObjectLock olock(newConfigPart.UpdateV1);
|
||||
for (const Dictionary::Pair& kv : newConfigPart.UpdateV1) {
|
||||
String path = "/" + zf.Tag + kv.first;
|
||||
|
||||
newConfigInfo.UpdateV1->Set(path, kv.second);
|
||||
newConfigInfo.Checksums->Set(path, GetChecksum(kv.second));
|
||||
}
|
||||
}
|
||||
|
||||
// Meta files.
|
||||
{
|
||||
ObjectLock olock(newConfigPart.UpdateV2);
|
||||
for (const Dictionary::Pair& kv : newConfigPart.UpdateV2) {
|
||||
String path = "/" + zf.Tag + kv.first;
|
||||
|
||||
newConfigInfo.UpdateV2->Set(path, kv.second);
|
||||
newConfigInfo.Checksums->Set(path, GetChecksum(kv.second));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
size_t sumUpdates = newConfigInfo.UpdateV1->GetLength() + newConfigInfo.UpdateV2->GetLength();
|
||||
|
||||
// Return early if there are no updates.
|
||||
if (sumUpdates == 0)
|
||||
return;
|
||||
|
||||
String productionZonesDir = GetApiZonesDir() + zoneName;
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Copying " << sumUpdates << " zone configuration files for zone '" << zoneName << "' to '" << productionZonesDir << "'.";
|
||||
|
||||
// Purge files to allow deletion via zones.d.
|
||||
if (Utility::PathExists(productionZonesDir))
|
||||
Utility::RemoveDirRecursive(productionZonesDir);
|
||||
|
||||
Utility::MkDirP(productionZonesDir, 0700);
|
||||
|
||||
// Copy content and add additional meta data.
|
||||
size_t numBytes = 0;
|
||||
|
||||
/* Note: We cannot simply copy directories here.
|
||||
*
|
||||
* Zone directories are registered from everywhere and we already
|
||||
* have read their content into memory with LoadConfigDir().
|
||||
*/
|
||||
Dictionary::Ptr newConfig = MergeConfigUpdate(newConfigInfo);
|
||||
|
||||
{
|
||||
ObjectLock olock(newConfig);
|
||||
|
||||
for (const Dictionary::Pair& kv : newConfig) {
|
||||
String dst = productionZonesDir + "/" + kv.first;
|
||||
|
||||
Utility::MkDirP(Utility::DirName(dst), 0755);
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Updating configuration file: " << dst;
|
||||
|
||||
String content = kv.second;
|
||||
|
||||
std::ofstream fp(dst.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
|
||||
|
||||
fp << content;
|
||||
fp.close();
|
||||
|
||||
numBytes += content.GetLength();
|
||||
}
|
||||
}
|
||||
|
||||
// Additional metadata.
|
||||
String tsPath = productionZonesDir + "/.timestamp";
|
||||
|
||||
if (!Utility::PathExists(tsPath)) {
|
||||
std::ofstream fp(tsPath.CStr(), std::ofstream::out | std::ostream::trunc);
|
||||
|
||||
fp << std::fixed << Utility::GetTime();
|
||||
fp.close();
|
||||
}
|
||||
|
||||
String authPath = productionZonesDir + "/.authoritative";
|
||||
|
||||
if (!Utility::PathExists(authPath)) {
|
||||
std::ofstream fp(authPath.CStr(), std::ofstream::out | std::ostream::trunc);
|
||||
fp.close();
|
||||
}
|
||||
|
||||
// Checksums.
|
||||
String checksumsPath = productionZonesDir + "/.checksums";
|
||||
|
||||
if (Utility::PathExists(checksumsPath))
|
||||
Utility::Remove(checksumsPath);
|
||||
|
||||
std::ofstream fp(checksumsPath.CStr(), std::ofstream::out | std::ostream::trunc);
|
||||
|
||||
fp << std::fixed << JsonEncode(newConfigInfo.Checksums);
|
||||
fp.close();
|
||||
|
||||
Log(LogNotice, "ApiListener")
|
||||
<< "Updated meta data for cluster config sync. Checksum: '" << checksumsPath
|
||||
<< "', timestamp: '" << tsPath << "', auth: '" << authPath << "'.";
|
||||
}
|
||||
|
||||
/**
|
||||
* Entrypoint for sending a file based config update to a cluster client.
|
||||
* This includes security checks for zone relations.
|
||||
* Loads the zone config files where this client belongs to
|
||||
* and sends the 'config::Update' JSON-RPC message.
|
||||
*
|
||||
* @param aclient Connected JSON-RPC client.
|
||||
*/
|
||||
void ApiListener::SendConfigUpdate(const JsonRpcConnection::Ptr& aclient)
|
||||
{
|
||||
Endpoint::Ptr endpoint = aclient->GetEndpoint();
|
||||
ASSERT(endpoint);
|
||||
|
||||
Zone::Ptr azone = endpoint->GetZone();
|
||||
Zone::Ptr lzone = Zone::GetLocalZone();
|
||||
Zone::Ptr clientZone = endpoint->GetZone();
|
||||
Zone::Ptr localZone = Zone::GetLocalZone();
|
||||
|
||||
/* don't try to send config updates to our master */
|
||||
if (!azone->IsChildOf(lzone))
|
||||
// Don't send config updates to parent zones
|
||||
if (!clientZone->IsChildOf(localZone))
|
||||
return;
|
||||
|
||||
Dictionary::Ptr configUpdateV1 = new Dictionary();
|
||||
Dictionary::Ptr configUpdateV2 = new Dictionary();
|
||||
Dictionary::Ptr configUpdateChecksums = new Dictionary(); // new since 2.11
|
||||
|
||||
String zonesDir = Configuration::DataDir + "/api/zones";
|
||||
String zonesDir = GetApiZonesDir();
|
||||
|
||||
for (const Zone::Ptr& zone : ConfigType::GetObjectsByType<Zone>()) {
|
||||
String zoneDir = zonesDir + "/" + zone->GetName();
|
||||
String zoneName = zone->GetName();
|
||||
String zoneDir = zonesDir + zoneName;
|
||||
|
||||
if (!zone->IsChildOf(azone) && !zone->IsGlobal())
|
||||
// Only sync child and global zones.
|
||||
if (!zone->IsChildOf(clientZone) && !zone->IsGlobal())
|
||||
continue;
|
||||
|
||||
// Zone was configured, but there's no configuration directory.
|
||||
if (!Utility::PathExists(zoneDir))
|
||||
continue;
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Syncing configuration files for " << (zone->IsGlobal() ? "global " : "")
|
||||
<< "zone '" << zone->GetName() << "' to endpoint '" << endpoint->GetName() << "'.";
|
||||
<< "zone '" << zoneName << "' to endpoint '" << endpoint->GetName() << "'.";
|
||||
|
||||
ConfigDirInformation config = LoadConfigDir(zonesDir + "/" + zone->GetName());
|
||||
configUpdateV1->Set(zone->GetName(), config.UpdateV1);
|
||||
configUpdateV2->Set(zone->GetName(), config.UpdateV2);
|
||||
ConfigDirInformation config = LoadConfigDir(zoneDir);
|
||||
|
||||
configUpdateV1->Set(zoneName, config.UpdateV1);
|
||||
configUpdateV2->Set(zoneName, config.UpdateV2);
|
||||
configUpdateChecksums->Set(zoneName, config.Checksums); // new since 2.11
|
||||
}
|
||||
|
||||
Dictionary::Ptr message = new Dictionary({
|
||||
|
|
@ -251,15 +221,29 @@ void ApiListener::SendConfigUpdate(const JsonRpcConnection::Ptr& aclient)
|
|||
{ "method", "config::Update" },
|
||||
{ "params", new Dictionary({
|
||||
{ "update", configUpdateV1 },
|
||||
{ "update_v2", configUpdateV2 }
|
||||
{ "update_v2", configUpdateV2 }, // Since 2.4.2.
|
||||
{ "checksums", configUpdateChecksums } // Since 2.11.0.
|
||||
}) }
|
||||
});
|
||||
|
||||
aclient->SendMessage(message);
|
||||
}
|
||||
|
||||
/**
|
||||
* Registered handler when a new config::Update message is received.
|
||||
*
|
||||
* Checks destination and permissions first, locks the transaction and analyses the update.
|
||||
* The newly received configuration is not copied to production immediately,
|
||||
* but into the staging directory first.
|
||||
* Last, the async validation and restart is triggered.
|
||||
*
|
||||
* @param origin Where this message came from.
|
||||
* @param params Message parameters including the config updates.
|
||||
* @returns Empty, required by the interface.
|
||||
*/
|
||||
Value ApiListener::ConfigUpdateHandler(const MessageOrigin::Ptr& origin, const Dictionary::Ptr& params)
|
||||
{
|
||||
// Verify permissions and trust relationship.
|
||||
if (!origin->FromClient->GetEndpoint() || (origin->FromZone && !Zone::GetLocalZone()->IsChildOf(origin->FromZone)))
|
||||
return Empty;
|
||||
|
||||
|
|
@ -281,52 +265,526 @@ Value ApiListener::ConfigUpdateHandler(const MessageOrigin::Ptr& origin, const D
|
|||
*/
|
||||
boost::mutex::scoped_lock lock(m_ConfigSyncStageLock);
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Applying config update from endpoint '" << origin->FromClient->GetEndpoint()->GetName()
|
||||
<< "' of zone '" << GetFromZoneName(origin->FromZone) << "'.";
|
||||
String apiZonesStageDir = GetApiZonesStageDir();
|
||||
String fromEndpointName = origin->FromClient->GetEndpoint()->GetName();
|
||||
String fromZoneName = GetFromZoneName(origin->FromZone);
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Applying config update from endpoint '" << fromEndpointName
|
||||
<< "' of zone '" << fromZoneName << "'.";
|
||||
|
||||
// Config files.
|
||||
Dictionary::Ptr updateV1 = params->Get("update");
|
||||
// Meta data files: .timestamp, etc.
|
||||
Dictionary::Ptr updateV2 = params->Get("update_v2");
|
||||
|
||||
// New since 2.11.0.
|
||||
Dictionary::Ptr checksums;
|
||||
|
||||
if (params->Contains("checksums"))
|
||||
checksums = params->Get("checksums");
|
||||
|
||||
bool configChange = false;
|
||||
|
||||
// Keep track of the relative config paths for later validation and copying. TODO: Find a better algorithm.
|
||||
std::vector<String> relativePaths;
|
||||
|
||||
/*
|
||||
* We can and must safely purge the staging directory, as the difference is taken between
|
||||
* runtime production config and newly received configuration.
|
||||
* This is needed to not mix deleted/changed content between received and stage
|
||||
* config.
|
||||
*/
|
||||
if (Utility::PathExists(apiZonesStageDir))
|
||||
Utility::RemoveDirRecursive(apiZonesStageDir);
|
||||
|
||||
Utility::MkDirP(apiZonesStageDir, 0700);
|
||||
|
||||
// Analyse and process the update.
|
||||
ObjectLock olock(updateV1);
|
||||
|
||||
for (const Dictionary::Pair& kv : updateV1) {
|
||||
Zone::Ptr zone = Zone::GetByName(kv.first);
|
||||
|
||||
// Check for the configured zones.
|
||||
String zoneName = kv.first;
|
||||
Zone::Ptr zone = Zone::GetByName(zoneName);
|
||||
|
||||
if (!zone) {
|
||||
Log(LogWarning, "ApiListener")
|
||||
<< "Ignoring config update for unknown zone '" << kv.first << "'.";
|
||||
<< "Ignoring config update from endpoint '" << fromEndpointName
|
||||
<< "' for unknown zone '" << zoneName << "'.";
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
if (ConfigCompiler::HasZoneConfigAuthority(kv.first)) {
|
||||
Log(LogWarning, "ApiListener")
|
||||
<< "Ignoring config update for zone '" << kv.first << "' because we have an authoritative version of the zone's config.";
|
||||
// Ignore updates where we have an authoritive copy in etc/zones.d, packages, etc.
|
||||
if (ConfigCompiler::HasZoneConfigAuthority(zoneName)) {
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Ignoring config update from endpoint '" << fromEndpointName
|
||||
<< "' for zone '" << zoneName << "' because we have an authoritative version of the zone's config.";
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
String oldDir = Configuration::DataDir + "/api/zones/" + zone->GetName();
|
||||
// Put the received configuration into our stage directory.
|
||||
String productionConfigZoneDir = GetApiZonesDir() + zoneName;
|
||||
String stageConfigZoneDir = GetApiZonesStageDir() + zoneName;
|
||||
|
||||
Utility::MkDirP(oldDir, 0700);
|
||||
Utility::MkDirP(productionConfigZoneDir, 0700);
|
||||
Utility::MkDirP(stageConfigZoneDir, 0700);
|
||||
|
||||
// Merge the config information.
|
||||
ConfigDirInformation newConfigInfo;
|
||||
newConfigInfo.UpdateV1 = kv.second;
|
||||
|
||||
// Load metadata.
|
||||
if (updateV2)
|
||||
newConfigInfo.UpdateV2 = updateV2->Get(kv.first);
|
||||
|
||||
Dictionary::Ptr newConfig = kv.second;
|
||||
ConfigDirInformation oldConfigInfo = LoadConfigDir(oldDir);
|
||||
// Load checksums. New since 2.11.
|
||||
if (checksums)
|
||||
newConfigInfo.Checksums = checksums->Get(kv.first);
|
||||
|
||||
if (UpdateConfigDir(oldConfigInfo, newConfigInfo, oldDir, false))
|
||||
configChange = true;
|
||||
// Load the current production config details.
|
||||
ConfigDirInformation productionConfigInfo = LoadConfigDir(productionConfigZoneDir);
|
||||
|
||||
// Merge updateV1 and updateV2
|
||||
Dictionary::Ptr productionConfig = MergeConfigUpdate(productionConfigInfo);
|
||||
Dictionary::Ptr newConfig = MergeConfigUpdate(newConfigInfo);
|
||||
|
||||
/* If we have received 'checksums' via cluster message, go for it.
|
||||
* Otherwise do the old timestamp dance for versions < 2.11.
|
||||
*/
|
||||
if (checksums) {
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Received configuration for zone '" << zoneName << "' from endpoint '"
|
||||
<< fromEndpointName << "'. Comparing the checksums.";
|
||||
|
||||
// TODO: Do this earlier in hello-handshakes?
|
||||
if (CheckConfigChange(productionConfigInfo, newConfigInfo))
|
||||
configChange = true;
|
||||
|
||||
} else {
|
||||
/* Fallback to timestamp handling when the parent endpoint didn't send checks.
|
||||
* This can happen when the satellite is 2.11 and the master is 2.10.
|
||||
*
|
||||
* TODO: Deprecate and remove this behaviour in 2.13+.
|
||||
*/
|
||||
|
||||
Log(LogWarning, "ApiListener")
|
||||
<< "Received configuration update without checksums from parent endpoint "
|
||||
<< fromEndpointName << ". This behaviour is deprecated. Please upgrade the parent endpoint to 2.11+";
|
||||
|
||||
double productionTimestamp;
|
||||
|
||||
if (!productionConfig->Contains("/.timestamp"))
|
||||
productionTimestamp = 0;
|
||||
else
|
||||
productionTimestamp = productionConfig->Get("/.timestamp");
|
||||
|
||||
double newTimestamp;
|
||||
|
||||
if (!newConfig->Contains("/.timestamp"))
|
||||
newTimestamp = Utility::GetTime();
|
||||
else
|
||||
newTimestamp = newConfig->Get("/.timestamp");
|
||||
|
||||
// Skip update if our configuration files are more recent.
|
||||
if (productionTimestamp >= newTimestamp) {
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Our configuration is more recent than the received configuration update."
|
||||
<< " Ignoring configuration file update for path '" << stageConfigZoneDir << "'. Current timestamp '"
|
||||
<< Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", productionTimestamp) << "' ("
|
||||
<< std::fixed << std::setprecision(6) << productionTimestamp
|
||||
<< ") >= received timestamp '"
|
||||
<< Utility::FormatDateTime("%Y-%m-%d %H:%M:%S %z", newTimestamp) << "' ("
|
||||
<< newTimestamp << ").";
|
||||
|
||||
} else {
|
||||
configChange = true;
|
||||
}
|
||||
|
||||
// Keep another hack when there's a timestamp file missing.
|
||||
{
|
||||
ObjectLock olock(newConfig);
|
||||
|
||||
for (const Dictionary::Pair &kv : newConfig) {
|
||||
|
||||
// This is super expensive with a string content comparison.
|
||||
if (productionConfig->Get(kv.first) != kv.second) {
|
||||
if (!Utility::Match("*/.timestamp", kv.first))
|
||||
configChange = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Update the .timestamp file.
|
||||
String tsPath = stageConfigZoneDir + "/.timestamp";
|
||||
if (!Utility::PathExists(tsPath)) {
|
||||
std::ofstream fp(tsPath.CStr(), std::ofstream::out | std::ostream::trunc);
|
||||
fp << std::fixed << newTimestamp;
|
||||
fp.close();
|
||||
}
|
||||
}
|
||||
|
||||
// Dump the received configuration for this zone into the stage directory.
|
||||
size_t numBytes = 0;
|
||||
|
||||
{
|
||||
ObjectLock olock(newConfig);
|
||||
|
||||
for (const Dictionary::Pair& kv : newConfig) {
|
||||
|
||||
/* Store the relative config file path for later validation and activation.
|
||||
* IMPORTANT: Store this prior to any filters.
|
||||
* */
|
||||
relativePaths.push_back(zoneName + "/" + kv.first);
|
||||
|
||||
// Ignore same config content. This is an expensive comparison.
|
||||
if (productionConfig->Get(kv.first) == kv.second)
|
||||
continue;
|
||||
|
||||
String path = stageConfigZoneDir + "/" + kv.first;
|
||||
|
||||
if (Utility::Match("*.conf", path)) {
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Stage: Updating received configuration file '" << path << "' for zone '" << zoneName << "'.";
|
||||
}
|
||||
|
||||
// Sync string content only.
|
||||
String content = kv.second;
|
||||
|
||||
// Generate a directory tree (zones/1/2/3 might not exist yet).
|
||||
Utility::MkDirP(Utility::DirName(path), 0755);
|
||||
|
||||
// Write the content to file.
|
||||
std::ofstream fp(path.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
|
||||
fp << content;
|
||||
fp.close();
|
||||
|
||||
numBytes += content.GetLength();
|
||||
}
|
||||
}
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Applying configuration file update for path '" << stageConfigZoneDir << "' ("
|
||||
<< numBytes << " Bytes).";
|
||||
|
||||
// If the update removes a path, delete it on disk and signal a config change.
|
||||
{
|
||||
ObjectLock xlock(productionConfig);
|
||||
|
||||
for (const Dictionary::Pair& kv : productionConfig) {
|
||||
if (!newConfig->Contains(kv.first)) {
|
||||
configChange = true;
|
||||
|
||||
String path = stageConfigZoneDir + "/" + kv.first;
|
||||
Utility::Remove(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* We have processed all configuration files and stored them in the staging directory.
|
||||
*
|
||||
* We need to store them locally for later analysis. A config change means
|
||||
* that we will validate the configuration in a separate process sandbox,
|
||||
* and only copy the configuration to production when everything is ok.
|
||||
*
|
||||
* A successful validation also triggers the final restart.
|
||||
*/
|
||||
if (configChange) {
|
||||
Log(LogInformation, "ApiListener", "Restarting after configuration change.");
|
||||
Application::RequestRestart();
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Received configuration from endpoint '" << fromEndpointName
|
||||
<< "' is different to production, triggering validation and reload.";
|
||||
AsyncTryActivateZonesStage(relativePaths);
|
||||
} else {
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Received configuration from endpoint '" << fromEndpointName
|
||||
<< "' is equal to production, not triggering reload.";
|
||||
}
|
||||
|
||||
return Empty;
|
||||
}
|
||||
|
||||
/**
|
||||
* Callback for stage config validation.
|
||||
* When validation was successful, the configuration is copied from
|
||||
* stage to production and a restart is triggered.
|
||||
* On failure, there's no restart and this is logged.
|
||||
*
|
||||
* @param pr Result of the validation process.
|
||||
* @param relativePaths Collected paths including the zone name, which are copied from stage to current directories.
|
||||
*/
|
||||
void ApiListener::TryActivateZonesStageCallback(const ProcessResult& pr,
|
||||
const std::vector<String>& relativePaths)
|
||||
{
|
||||
String apiZonesDir = GetApiZonesDir();
|
||||
String apiZonesStageDir = GetApiZonesStageDir();
|
||||
|
||||
String logFile = apiZonesStageDir + "/startup.log";
|
||||
std::ofstream fpLog(logFile.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
|
||||
fpLog << pr.Output;
|
||||
fpLog.close();
|
||||
|
||||
String statusFile = apiZonesStageDir + "/status";
|
||||
std::ofstream fpStatus(statusFile.CStr(), std::ofstream::out | std::ostream::binary | std::ostream::trunc);
|
||||
fpStatus << pr.ExitStatus;
|
||||
fpStatus.close();
|
||||
|
||||
// Validation went fine, copy stage and reload.
|
||||
if (pr.ExitStatus == 0) {
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Config validation for stage '" << apiZonesStageDir << "' was OK, replacing into '" << apiZonesDir << "' and triggering reload.";
|
||||
|
||||
// Purge production before copying stage.
|
||||
if (Utility::PathExists(apiZonesDir))
|
||||
Utility::RemoveDirRecursive(apiZonesDir);
|
||||
|
||||
Utility::MkDirP(apiZonesDir, 0700);
|
||||
|
||||
// Copy all synced configuration files from stage to production.
|
||||
for (const String& path : relativePaths) {
|
||||
if (!Utility::PathExists(path))
|
||||
continue;
|
||||
|
||||
Log(LogInformation, "ApiListener")
|
||||
<< "Copying file '" << path << "' from config sync staging to production zones directory.";
|
||||
|
||||
String stagePath = apiZonesStageDir + path;
|
||||
String currentPath = apiZonesDir + path;
|
||||
|
||||
Utility::MkDirP(Utility::DirName(currentPath), 0700);
|
||||
|
||||
Utility::CopyFile(stagePath, currentPath);
|
||||
}
|
||||
|
||||
// Clear any failed deployment before
|
||||
ApiListener::Ptr listener = ApiListener::GetInstance();
|
||||
|
||||
if (listener)
|
||||
listener->ClearLastFailedZonesStageValidation();
|
||||
|
||||
Application::RequestRestart();
|
||||
|
||||
// All good, return early.
|
||||
return;
|
||||
}
|
||||
|
||||
// Error case.
|
||||
Log(LogCritical, "ApiListener")
|
||||
<< "Config validation failed for staged cluster config sync in '" << apiZonesStageDir
|
||||
<< "'. Aborting. Logs: '" << logFile << "'";
|
||||
|
||||
ApiListener::Ptr listener = ApiListener::GetInstance();
|
||||
|
||||
if (listener)
|
||||
listener->UpdateLastFailedZonesStageValidation(pr.Output);
|
||||
}
|
||||
|
||||
/**
|
||||
* Spawns a new validation process and waits for its output.
|
||||
* Sets 'System.ZonesStageVarDir' to override the config validation zone dirs with our current stage.
|
||||
*
|
||||
* @param relativePaths Required for later file operations in the callback. Provides the zone name plus path in a list.
|
||||
*/
|
||||
void ApiListener::AsyncTryActivateZonesStage(const std::vector<String>& relativePaths)
|
||||
{
|
||||
VERIFY(Application::GetArgC() >= 1);
|
||||
|
||||
/* Inherit parent process args. */
|
||||
Array::Ptr args = new Array({
|
||||
Application::GetExePath(Application::GetArgV()[0]),
|
||||
});
|
||||
|
||||
for (int i = 1; i < Application::GetArgC(); i++) {
|
||||
String argV = Application::GetArgV()[i];
|
||||
|
||||
if (argV == "-d" || argV == "--daemonize")
|
||||
continue;
|
||||
|
||||
args->Add(argV);
|
||||
}
|
||||
|
||||
args->Add("--validate");
|
||||
|
||||
// Set the ZonesStageDir. This creates our own local chroot without any additional automated zone includes.
|
||||
args->Add("--define");
|
||||
args->Add("System.ZonesStageVarDir=" + GetApiZonesStageDir());
|
||||
|
||||
Process::Ptr process = new Process(Process::PrepareCommand(args));
|
||||
process->SetTimeout(Application::GetReloadTimeout());
|
||||
process->Run(std::bind(&TryActivateZonesStageCallback, _1, relativePaths));
|
||||
}
|
||||
|
||||
/**
|
||||
* Update the structure from the last failed validation output.
|
||||
* Uses the current timestamp.
|
||||
*
|
||||
* @param log The process output from the config validation.
|
||||
*/
|
||||
void ApiListener::UpdateLastFailedZonesStageValidation(const String& log)
|
||||
{
|
||||
Dictionary::Ptr lastFailedZonesStageValidation = new Dictionary({
|
||||
{ "log", log },
|
||||
{ "ts", Utility::GetTime() }
|
||||
});
|
||||
|
||||
SetLastFailedZonesStageValidation(lastFailedZonesStageValidation);
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear the structure for the last failed reload.
|
||||
*
|
||||
*/
|
||||
void ApiListener::ClearLastFailedZonesStageValidation()
|
||||
{
|
||||
SetLastFailedZonesStageValidation(Dictionary::Ptr());
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate a config checksum.
|
||||
*
|
||||
* @param content String content used for generating the checksum.
|
||||
* @returns The checksum as string.
|
||||
*/
|
||||
String ApiListener::GetChecksum(const String& content)
|
||||
{
|
||||
return SHA256(content);
|
||||
}
|
||||
|
||||
bool ApiListener::CheckConfigChange(const ConfigDirInformation& oldConfig, const ConfigDirInformation& newConfig)
|
||||
{
|
||||
Dictionary::Ptr oldChecksums = oldConfig.Checksums;
|
||||
Dictionary::Ptr newChecksums = newConfig.Checksums;
|
||||
|
||||
// TODO: Figure out whether normal users need this for debugging.
|
||||
Log(LogDebug, "ApiListener")
|
||||
<< "Checking for config change between stage and production. Old (" << oldChecksums->GetLength() << "): '"
|
||||
<< JsonEncode(oldChecksums)
|
||||
<< "' vs. new (" << newChecksums->GetLength() << "): '"
|
||||
<< JsonEncode(newChecksums) << "'.";
|
||||
|
||||
// Different length means that either one or the other side added or removed something. */
|
||||
if (oldChecksums->GetLength() != newChecksums->GetLength())
|
||||
return true;
|
||||
|
||||
// Both dictionaries have an equal size.
|
||||
ObjectLock olock(oldChecksums);
|
||||
|
||||
for (const Dictionary::Pair& kv : oldChecksums) {
|
||||
String path = kv.first;
|
||||
String oldChecksum = kv.second;
|
||||
|
||||
// TODO: Figure out if config changes only apply to '.conf'. Leaving this open for other config files.
|
||||
//if (!Utility::Match("*.conf", path))
|
||||
// continue;
|
||||
|
||||
/* Ignore internal files, especially .timestamp and .checksums.
|
||||
*
|
||||
* If we don't, this results in "always change" restart loops.
|
||||
*/
|
||||
if (Utility::Match("/.*", path))
|
||||
continue;
|
||||
|
||||
Log(LogDebug, "ApiListener")
|
||||
<< "Checking " << path << " for checksum: " << oldChecksum;
|
||||
|
||||
// Check whether our key exists in the new checksums, and they have an equal value.
|
||||
String newChecksum = newChecksums->Get(path);
|
||||
|
||||
if (newChecksums->Get(path) != kv.second) {
|
||||
Log(LogDebug, "ApiListener")
|
||||
<< "Path '" << path << "' doesn't match old checksum '"
|
||||
<< newChecksum << "' with new checksum '" << oldChecksum << "'.";
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Load the given config dir and read their file content into the config structure.
|
||||
*
|
||||
* @param dir Path to the config directory.
|
||||
* @returns ConfigDirInformation structure.
|
||||
*/
|
||||
ConfigDirInformation ApiListener::LoadConfigDir(const String& dir)
|
||||
{
|
||||
ConfigDirInformation config;
|
||||
config.UpdateV1 = new Dictionary();
|
||||
config.UpdateV2 = new Dictionary();
|
||||
config.Checksums = new Dictionary();
|
||||
|
||||
Utility::GlobRecursive(dir, "*", std::bind(&ApiListener::ConfigGlobHandler, std::ref(config), dir, _1), GlobFile);
|
||||
return config;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the given file and store it in the config information structure.
|
||||
* Callback function for Glob().
|
||||
*
|
||||
* @param config Reference to the config information object.
|
||||
* @param path File path.
|
||||
* @param file Full file name.
|
||||
*/
|
||||
void ApiListener::ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file)
|
||||
{
|
||||
// Avoid loading the authoritative marker for syncs at all cost.
|
||||
if (Utility::BaseName(file) == ".authoritative")
|
||||
return;
|
||||
|
||||
CONTEXT("Creating config update for file '" + file + "'");
|
||||
|
||||
Log(LogNotice, "ApiListener")
|
||||
<< "Creating config update for file '" << file << "'.";
|
||||
|
||||
std::ifstream fp(file.CStr(), std::ifstream::binary);
|
||||
if (!fp)
|
||||
return;
|
||||
|
||||
String content((std::istreambuf_iterator<char>(fp)), std::istreambuf_iterator<char>());
|
||||
|
||||
Dictionary::Ptr update;
|
||||
String relativePath = file.SubStr(path.GetLength());
|
||||
|
||||
/*
|
||||
* 'update' messages contain conf files. 'update_v2' syncs everything else (.timestamp).
|
||||
*
|
||||
* **Keep this intact to stay compatible with older clients.**
|
||||
*/
|
||||
if (Utility::Match("*.conf", file))
|
||||
update = config.UpdateV1;
|
||||
else
|
||||
update = config.UpdateV2;
|
||||
|
||||
update->Set(relativePath, content);
|
||||
|
||||
/* Calculate a checksum for each file (and a global one later).
|
||||
*
|
||||
* IMPORTANT: Ignore the .authoritative file above, this must not be synced.
|
||||
* */
|
||||
config.Checksums->Set(relativePath, GetChecksum(content));
|
||||
}
|
||||
|
||||
/**
|
||||
* Compatibility helper for merging config update v1 and v2 into a global result.
|
||||
*
|
||||
* @param config Config information structure.
|
||||
* @returns Dictionary which holds the merged information.
|
||||
*/
|
||||
Dictionary::Ptr ApiListener::MergeConfigUpdate(const ConfigDirInformation& config)
|
||||
{
|
||||
Dictionary::Ptr result = new Dictionary();
|
||||
|
||||
if (config.UpdateV1)
|
||||
config.UpdateV1->CopyTo(result);
|
||||
|
||||
if (config.UpdateV2)
|
||||
config.UpdateV2->CopyTo(result);
|
||||
|
||||
return result;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -59,6 +59,16 @@ String ApiListener::GetApiDir()
|
|||
return Configuration::DataDir + "/api/";
|
||||
}
|
||||
|
||||
String ApiListener::GetApiZonesDir()
|
||||
{
|
||||
return GetApiDir() + "zones/";
|
||||
}
|
||||
|
||||
String ApiListener::GetApiZonesStageDir()
|
||||
{
|
||||
return GetApiDir() + "zones-stage/";
|
||||
}
|
||||
|
||||
String ApiListener::GetCertsDir()
|
||||
{
|
||||
return Configuration::DataDir + "/certs/";
|
||||
|
|
@ -232,7 +242,7 @@ void ApiListener::Start(bool runtimeCreated)
|
|||
Log(LogInformation, "ApiListener")
|
||||
<< "'" << GetName() << "' started.";
|
||||
|
||||
SyncZoneDirs();
|
||||
SyncLocalZoneDirs();
|
||||
|
||||
ObjectImpl<ApiListener>::Start(runtimeCreated);
|
||||
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@
|
|||
#include "remote/endpoint.hpp"
|
||||
#include "remote/messageorigin.hpp"
|
||||
#include "base/configobject.hpp"
|
||||
#include "base/process.hpp"
|
||||
#include "base/timer.hpp"
|
||||
#include "base/workqueue.hpp"
|
||||
#include "base/tcpsocket.hpp"
|
||||
|
|
@ -31,6 +32,7 @@ struct ConfigDirInformation
|
|||
{
|
||||
Dictionary::Ptr UpdateV1;
|
||||
Dictionary::Ptr UpdateV2;
|
||||
Dictionary::Ptr Checksums;
|
||||
};
|
||||
|
||||
/**
|
||||
|
|
@ -47,6 +49,8 @@ public:
|
|||
ApiListener();
|
||||
|
||||
static String GetApiDir();
|
||||
static String GetApiZonesDir();
|
||||
static String GetApiZonesStageDir();
|
||||
static String GetCertsDir();
|
||||
static String GetCaDir();
|
||||
static String GetCertificateRequestsDir();
|
||||
|
|
@ -167,16 +171,26 @@ private:
|
|||
/* filesync */
|
||||
static boost::mutex m_ConfigSyncStageLock;
|
||||
|
||||
static ConfigDirInformation LoadConfigDir(const String& dir);
|
||||
static Dictionary::Ptr MergeConfigUpdate(const ConfigDirInformation& config);
|
||||
static bool UpdateConfigDir(const ConfigDirInformation& oldConfig, const ConfigDirInformation& newConfig, const String& configDir, bool authoritative);
|
||||
void SyncLocalZoneDirs() const;
|
||||
void SyncLocalZoneDir(const Zone::Ptr& zone) const;
|
||||
|
||||
void SyncZoneDirs() const;
|
||||
void SyncZoneDir(const Zone::Ptr& zone) const;
|
||||
|
||||
static void ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file);
|
||||
void SendConfigUpdate(const JsonRpcConnection::Ptr& aclient);
|
||||
|
||||
static Dictionary::Ptr MergeConfigUpdate(const ConfigDirInformation& config);
|
||||
|
||||
static ConfigDirInformation LoadConfigDir(const String& dir);
|
||||
static void ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file);
|
||||
|
||||
static void TryActivateZonesStageCallback(const ProcessResult& pr,
|
||||
const std::vector<String>& relativePaths);
|
||||
static void AsyncTryActivateZonesStage(const std::vector<String>& relativePaths);
|
||||
|
||||
static String GetChecksum(const String& content);
|
||||
static bool CheckConfigChange(const ConfigDirInformation& oldConfig, const ConfigDirInformation& newConfig);
|
||||
|
||||
void UpdateLastFailedZonesStageValidation(const String& log);
|
||||
void ClearLastFailedZonesStageValidation();
|
||||
|
||||
/* configsync */
|
||||
void UpdateConfigObject(const ConfigObject::Ptr& object, const MessageOrigin::Ptr& origin,
|
||||
const JsonRpcConnection::Ptr& client = nullptr);
|
||||
|
|
|
|||
|
|
@ -54,6 +54,8 @@ class ApiListener : ConfigObject
|
|||
[state, no_user_modify] Timestamp log_message_timestamp;
|
||||
|
||||
[no_user_modify] String identity;
|
||||
|
||||
[state, no_user_modify] Dictionary::Ptr last_failed_zones_stage_validation;
|
||||
};
|
||||
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue