diff --git a/doc/09-object-types.md b/doc/09-object-types.md index efe796e5c..6fa51301f 100644 --- a/doc/09-object-types.md +++ b/doc/09-object-types.md @@ -1727,6 +1727,9 @@ Configuration Attributes: host | String | **Optional.** OpenTSDB host address. Defaults to `127.0.0.1`. port | Number | **Optional.** OpenTSDB port. Defaults to `4242`. enable\_ha | Boolean | **Optional.** Enable the high availability functionality. Only valid in a [cluster setup](06-distributed-monitoring.md#distributed-monitoring-high-availability-features). Defaults to `false`. + enable_generic_metrics | Boolean | **Optional.** Re-use metric names to store different perfdata values for a particular check. Use tags to distinguish perfdata instead of metric name. Defaults to `false`. + host_template | Dictionary | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`. + service_template | Dictionary | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). Defaults to an `empty Dictionary`. ### PerfdataWriter diff --git a/doc/14-features.md b/doc/14-features.md index 4b56737bf..47402a340 100644 --- a/doc/14-features.md +++ b/doc/14-features.md @@ -618,20 +618,33 @@ You can enable the feature using By default the `OpenTsdbWriter` object expects the TSD to listen at `127.0.0.1` on port `4242`. -The current naming schema is +The current default naming schema is: ``` -icinga.host. -icinga.service.. +icinga.host. +icinga.service.. ``` -for host and service checks. The tag host is always applied. +for host and service checks. The tag `host` is always applied. + +Icinga also sends perfdata warning, critical, minimum and maximum threshold values to OpenTSDB. +These are stored as new OpenTSDB metric names appended with `_warn`, `_crit`, `_min`, `_max`. +Values are only stored when the corresponding threshold exists in Icinga's perfdata. + +Example: +``` +icinga.service.. +icinga.service..._warn +icinga.service..._crit +icinga.service..._min +icinga.service..._max +``` To make sure Icinga 2 writes a valid metric into OpenTSDB some characters are replaced with `_` in the target name: ``` -\ (and space) +\ : (and space) ``` The resulting name in OpenTSDB might look like: @@ -676,6 +689,167 @@ with the following tags > You might want to set the tsd.core.auto_create_metrics setting to `true` > in your opentsdb.conf configuration file. +#### OpenTSDB Metric Prefix +Functionality exists to modify the built in OpenTSDB metric names that the plugin +writes to. By default this is `icinga.host` and `icinga.service.`. + +These prefixes can be modified as necessary to any arbitary string. The prefix +configuration also supports Icinga macros, so if you rather use `` +or any other variable instead of `` you may do so. + +To configure OpenTSDB metric name prefixes, create or modify the `host_template` and/or +`service_template` blocks in the `opentsdb.conf` file, to add a `metric` definition. +These modifications go hand in hand with the **OpenTSDB Custom Tag Support** detailed below, +and more information around macro use can be found there. + +Additionally, using custom Metric Prefixes or your own macros in the prefix may be +helpful if you are using the **OpenTSDB Generic Metric** functionality detailed below. + +An example configuration which includes prefix name modification: + +``` +object OpenTsdbWriter "opentsdb" { + host = "127.0.0.1" + port = 4242 + host_template = { + metric = "icinga.myhost" + tags = { + location = "$host.vars.location$" + checkcommand = "$host.check_command$" + } + } + service_template = { + metric = "icinga.service.$service.check_command$" + } +} +``` + +The above configuration will output the following naming schema: +``` +icinga.myhost. +icinga.service.. +``` +Note how `` is always appended in the default naming schema mode. + +#### OpenTSDB Generic Metric Naming Schema + +An alternate naming schema (`Generic Metrics`) is available where OpenTSDB metric names are more generic +and do not include the Icinga perfdata label in the metric name. Instead, +perfdata labels are stored in a tag `label` which is stored along with each perfdata value. + +This ultimately reduces the number of unique OpenTSDB metric names which may make +querying aggregate data easier. This also allows you to store all perfdata values for a +particular check inside one OpenTSDB metric name for each check. + +This alternate naming schema can be enabled by setting the following in the OpenTSDBWriter config: +`enable_generic_metrics = true` + +> **Tip** +> Consider using `Generic Metrics` along with the **OpenTSDB Metric Prefix** naming options +> described above + +An example of this naming schema when compared to the default is: + +``` +icinga.host +icinga.service. +``` + +> **Note** +> Note how `` does not appear in the OpenTSDB metric name +> when using `Generic Metrics`. Instead, a new tag `label` appears on each value written +> to OpenTSDB which contains the perfdata label. + +#### Custom Tags + +In addition to the default tags listed above, it is possible to send +your own custom tags with your data to OpenTSDB. + +Note that custom tags are sent **in addition** to the default hostname, +type and service name tags. If you do not include this section in the +config file, no custom tags will be included. + +Custom tags can be custom attributes or built in attributes. + +Consider a host object: + +``` +object Host "my-server1" { + address = "10.0.0.1" + check_command = "hostalive" + vars.location = "Australia" +} +``` + +and a service object: + +``` +object Service "ping" { + host_name = "localhost" + check_command = "my-ping" + + vars.ping_packets = 10 +} +``` + +It is possible to send `vars.location` and `vars.ping_packets` along +with performance data. Additionally, any other attribute can be sent +as a tag, such as `check_command`. + +You can make use of the `host_template` and `service_template` blocks +in the `opentsdb.conf` configuration file. + +An example OpenTSDB configuration file which makes use of custom tags: + +``` +object OpenTsdbWriter "opentsdb" { + host = "127.0.0.1" + port = 4242 + host_template = { + tags = { + location = "$host.vars.location$" + checkcommand = "$host.check_command$" + } + } + service_template = { + tags = { + location = "$host.vars.location$" + pingpackets = "$service.vars.ping_packets$" + checkcommand = "$service.check_command$" + } + } +} +``` + +Depending on what keyword the macro begins with, will determine what +attributes are available in the macro context. The below table explains +what attributes are available with links to each object type. + + start of macro | description + ---------------|------------------------------------------ + \$host...$ | Attributes available on a [Host object](09-object-types.md#objecttype-host) + \$service...$ | Attributes available on a [Service object](09-object-types.md#objecttype-service) + \$icinga...$ | Attributes available on the [IcingaApplication object](09-object-types.md#icingaapplication) + +> **Note** +> +> Ensure you do not name your custom attributes with a dot in the name. +> Dots located inside a macro tell the interpreter to expand a +> dictionary. +> +> Do not do this in your object configuration: +> +> `vars["my.attribute"]` +> +> as you will be unable to reference `my.attribute` because it is not a +> dictionary. +> +> Instead, use underscores or another character: +> +> `vars.my_attribute` or `vars["my_attribute"]` + + + #### OpenTSDB in Cluster HA Zones The OpenTSDB feature supports [high availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-features) diff --git a/etc/icinga2/features-available/opentsdb.conf b/etc/icinga2/features-available/opentsdb.conf index 540556c10..471a62268 100644 --- a/etc/icinga2/features-available/opentsdb.conf +++ b/etc/icinga2/features-available/opentsdb.conf @@ -6,4 +6,20 @@ object OpenTsdbWriter "opentsdb" { //host = "127.0.0.1" //port = 4242 + //enable_generic_metrics = false + + // Custom Tagging, refer to Icinga object type documentation for + // OpenTsdbWriter + //host_template = { + // metric = "icinga.host" + // tags = { + // zone = "$host.zone$" + // } + //} + //service_template = { + // metric = "icinga.service.$service.check_command$" + // tags = { + // zone = "$service.zone$" + // } + //} } diff --git a/lib/perfdata/opentsdbwriter.cpp b/lib/perfdata/opentsdbwriter.cpp index 38405a2a3..125ecaaed 100644 --- a/lib/perfdata/opentsdbwriter.cpp +++ b/lib/perfdata/opentsdbwriter.cpp @@ -73,6 +73,8 @@ void OpenTsdbWriter::Resume() Log(LogInformation, "OpentsdbWriter") << "'" << GetName() << "' resumed."; + ReadConfigTemplate(m_ServiceConfigTemplate, m_HostConfigTemplate); + m_ReconnectTimer = new Timer(); m_ReconnectTimer->SetInterval(10); m_ReconnectTimer->OnTimerExpired.connect(std::bind(&OpenTsdbWriter::ReconnectTimerHandler, this)); @@ -151,28 +153,104 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C Service::Ptr service = dynamic_pointer_cast(checkable); Host::Ptr host; + Dictionary::Ptr config_tmpl; + Dictionary::Ptr config_tmpl_tags; + String config_tmpl_metric; - if (service) + if (service) { host = service->GetHost(); - else + config_tmpl = m_ServiceConfigTemplate; + } + else { host = static_pointer_cast(checkable); + config_tmpl = m_HostConfigTemplate; + } + + // Get the tags nested dictionary in the service/host template in the config + if (config_tmpl) { + config_tmpl_tags = config_tmpl->Get("tags"); + config_tmpl_metric = config_tmpl->Get("metric"); + } String metric; std::map tags; + // Resolve macros in configuration template and build custom tag list + if (config_tmpl_tags || !config_tmpl_metric.IsEmpty()) { + + // Configure config template macro resolver + MacroProcessor::ResolverList resolvers; + if (service) + resolvers.emplace_back("service", service); + resolvers.emplace_back("host", host); + resolvers.emplace_back("icinga", IcingaApplication::GetInstance()); + + // Resolve macros for the service and host template config line + if (config_tmpl_tags) { + ObjectLock olock(config_tmpl_tags); + + for (const Dictionary::Pair& pair : config_tmpl_tags) { + + String missing_macro; + Value value = MacroProcessor::ResolveMacros(pair.second, resolvers, cr, &missing_macro); + + if (!missing_macro.IsEmpty()) { + Log(LogDebug, "OpenTsdbWriter") + << "Unable to resolve macro:'" << missing_macro + << "' for this host or service."; + + continue; + } + + String tagname = Convert::ToString(pair.first); + tags[tagname] = EscapeTag(value); + + } + } + + // Resolve macros for the metric config line + if (!config_tmpl_metric.IsEmpty()) { + + String missing_macro; + Value value = MacroProcessor::ResolveMacros(config_tmpl_metric, resolvers, cr, &missing_macro); + + if (!missing_macro.IsEmpty()) { + Log(LogDebug, "OpenTsdbWriter") + << "Unable to resolve macro:'" << missing_macro + << "' for this host or service."; + + } + else { + + config_tmpl_metric = Convert::ToString(value); + + } + } + } + String escaped_hostName = EscapeTag(host->GetName()); tags["host"] = escaped_hostName; double ts = cr->GetExecutionEnd(); if (service) { - String serviceName = service->GetShortName(); - String escaped_serviceName = EscapeMetric(serviceName); - metric = "icinga.service." + escaped_serviceName; + if (!config_tmpl_metric.IsEmpty()) { + metric = config_tmpl_metric; + } else { + String serviceName = service->GetShortName(); + String escaped_serviceName = EscapeMetric(serviceName); + metric = "icinga.service." + escaped_serviceName; + } + SendMetric(checkable, metric + ".state", tags, service->GetState(), ts); + } else { - metric = "icinga.host"; + if (!config_tmpl_metric.IsEmpty()) { + metric = config_tmpl_metric; + } else { + metric = "icinga.host"; + } SendMetric(checkable, metric + ".state", tags, host->GetState(), ts); } @@ -236,20 +314,32 @@ void OpenTsdbWriter::SendPerfdata(const Checkable::Ptr& checkable, const String& continue; } } + + String metric_name; + std::map tags_new = tags; - String escaped_key = EscapeMetric(pdv->GetLabel()); - boost::algorithm::replace_all(escaped_key, "::", "."); + // Do not break original functionality where perfdata labels form + // part of the metric name + if (!GetEnableGenericMetrics()) { + String escaped_key = EscapeMetric(pdv->GetLabel()); + boost::algorithm::replace_all(escaped_key, "::", "."); + metric_name = metric + "." + escaped_key; + } else { + String escaped_key = EscapeTag(pdv->GetLabel()); + metric_name = metric; + tags_new["label"] = escaped_key; + } - SendMetric(checkable, metric + "." + escaped_key, tags, pdv->GetValue(), ts); + SendMetric(checkable, metric_name, tags_new, pdv->GetValue(), ts); if (pdv->GetCrit()) - SendMetric(checkable, metric + "." + escaped_key + "_crit", tags, pdv->GetCrit(), ts); + SendMetric(checkable, metric_name + "_crit", tags_new, pdv->GetCrit(), ts); if (pdv->GetWarn()) - SendMetric(checkable, metric + "." + escaped_key + "_warn", tags, pdv->GetWarn(), ts); + SendMetric(checkable, metric_name + "_warn", tags_new, pdv->GetWarn(), ts); if (pdv->GetMin()) - SendMetric(checkable, metric + "." + escaped_key + "_min", tags, pdv->GetMin(), ts); + SendMetric(checkable, metric_name + "_min", tags_new, pdv->GetMin(), ts); if (pdv->GetMax()) - SendMetric(checkable, metric + "." + escaped_key + "_max", tags, pdv->GetMax(), ts); + SendMetric(checkable, metric_name + "_max", tags_new, pdv->GetMax(), ts); } } @@ -316,6 +406,7 @@ String OpenTsdbWriter::EscapeTag(const String& str) boost::replace_all(result, " ", "_"); boost::replace_all(result, "\\", "_"); + boost::replace_all(result, ":", "_"); return result; } @@ -338,3 +429,86 @@ String OpenTsdbWriter::EscapeMetric(const String& str) return result; } + +/** +* Saves the template dictionaries defined in the config file into running memory +* +* @param stemplate The dictionary to save the service configuration to +* @param htemplate The dictionary to save the host configuration to +*/ +void OpenTsdbWriter::ReadConfigTemplate(const Dictionary::Ptr& stemplate, + const Dictionary::Ptr& htemplate) +{ + + m_ServiceConfigTemplate = GetServiceTemplate(); + + if (!m_ServiceConfigTemplate) { + Log(LogDebug, "OpenTsdbWriter") + << "Unable to locate service template configuration."; + } else if (m_ServiceConfigTemplate->GetLength() == 0) { + Log(LogDebug, "OpenTsdbWriter") + << "The service template configuration is empty."; + } + + m_HostConfigTemplate = GetHostTemplate(); + + if (!m_HostConfigTemplate) { + Log(LogDebug, "OpenTsdbWriter") + << "Unable to locate host template configuration."; + } else if (m_HostConfigTemplate->GetLength() == 0) { + Log(LogDebug, "OpenTsdbWriter") + << "The host template configuration is empty."; + } + +} + + +/** +* Validates the host_template configuration block in the configuration +* file and checks for syntax errors. +* +* @param lvalue The host_template dictionary +* @param utils Validation helper utilities +*/ +void OpenTsdbWriter::ValidateHostTemplate(const Lazy& lvalue, const ValidationUtils& utils) +{ + ObjectImpl::ValidateHostTemplate(lvalue, utils); + + String metric = lvalue()->Get("metric"); + if (!MacroProcessor::ValidateMacroString(metric)) + BOOST_THROW_EXCEPTION(ValidationError(this, { "host_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'.")); + + Dictionary::Ptr tags = lvalue()->Get("tags"); + if (tags) { + ObjectLock olock(tags); + for (const Dictionary::Pair& pair : tags) { + if (!MacroProcessor::ValidateMacroString(pair.second)) + BOOST_THROW_EXCEPTION(ValidationError(this, { "host_template", "tags", pair.first }, "Closing $ not found in macro format string '" + pair.second)); + } + } +} + +/** +* Validates the service_template configuration block in the +* configuration file and checks for syntax errors. +* +* @param lvalue The service_template dictionary +* @param utils Validation helper utilities +*/ +void OpenTsdbWriter::ValidateServiceTemplate(const Lazy& lvalue, const ValidationUtils& utils) +{ + ObjectImpl::ValidateServiceTemplate(lvalue, utils); + + String metric = lvalue()->Get("metric"); + if (!MacroProcessor::ValidateMacroString(metric)) + BOOST_THROW_EXCEPTION(ValidationError(this, { "service_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'.")); + + Dictionary::Ptr tags = lvalue()->Get("tags"); + if (tags) { + ObjectLock olock(tags); + for (const Dictionary::Pair& pair : tags) { + if (!MacroProcessor::ValidateMacroString(pair.second)) + BOOST_THROW_EXCEPTION(ValidationError(this, { "service_template", "tags", pair.first }, "Closing $ not found in macro format string '" + pair.second)); + } + } +} \ No newline at end of file diff --git a/lib/perfdata/opentsdbwriter.hpp b/lib/perfdata/opentsdbwriter.hpp index 5aff981cd..5531bdae2 100644 --- a/lib/perfdata/opentsdbwriter.hpp +++ b/lib/perfdata/opentsdbwriter.hpp @@ -26,6 +26,9 @@ public: static void StatsFunc(const Dictionary::Ptr& status, const Array::Ptr& perfdata); + void ValidateHostTemplate(const Lazy& lvalue, const ValidationUtils& utils) override; + void ValidateServiceTemplate(const Lazy& lvalue, const ValidationUtils& utils) override; + protected: void OnConfigLoaded() override; void Resume() override; @@ -36,6 +39,9 @@ private: Timer::Ptr m_ReconnectTimer; + Dictionary::Ptr m_ServiceConfigTemplate; + Dictionary::Ptr m_HostConfigTemplate; + void CheckResultHandler(const Checkable::Ptr& checkable, const CheckResult::Ptr& cr); void SendMetric(const Checkable::Ptr& checkable, const String& metric, const std::map& tags, double value, double ts); @@ -45,6 +51,9 @@ private: static String EscapeMetric(const String& str); void ReconnectTimerHandler(); + + void ReadConfigTemplate(const Dictionary::Ptr& stemplate, + const Dictionary::Ptr& htemplate); }; } diff --git a/lib/perfdata/opentsdbwriter.ti b/lib/perfdata/opentsdbwriter.ti index de19a1eac..626350a80 100644 --- a/lib/perfdata/opentsdbwriter.ti +++ b/lib/perfdata/opentsdbwriter.ti @@ -20,6 +20,16 @@ class OpenTsdbWriter : ConfigObject [config] bool enable_ha { default {{{ return false; }}} }; + [config] Dictionary::Ptr host_template { + default {{{ return new Dictionary(); }}} + + }; + [config] Dictionary::Ptr service_template { + default {{{ return new Dictionary(); }}} + }; + [config] bool enable_generic_metrics { + default {{{ return false; }}} + }; [no_user_modify] bool connected; [no_user_modify] bool should_connect { @@ -27,4 +37,19 @@ class OpenTsdbWriter : ConfigObject }; }; +validator OpenTsdbWriter { + Dictionary host_template { + String metric; + Dictionary "tags" { + String "*"; + }; + }; + Dictionary service_template { + String metric; + Dictionary "tags" { + String "*"; + }; + }; +}; + }