diff --git a/doc/09-object-types.md b/doc/09-object-types.md index f0d208030..9c8a73baf 100644 --- a/doc/09-object-types.md +++ b/doc/09-object-types.md @@ -1703,8 +1703,9 @@ Configuration Attributes: host | String | **Optional.** OpenTSDB host address. Defaults to `127.0.0.1`. port | Number | **Optional.** OpenTSDB port. Defaults to `4242`. enable\_ha | Boolean | **Optional.** Enable the high availability functionality. Only valid in a [cluster setup](06-distributed-monitoring.md#distributed-monitoring-high-availability-features). Defaults to `false`. - host_template | Dictionary | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`. - service_template | Dictionary | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`. + enable_generic_metrics | Boolean | **Optional.** Re-use metric names to store different perfdata values for a particular check. Use tags to distinguish perfdata instead of metric name. Defaults to `false`. + host_template | Dictionary | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`. + service_template | Dictionary | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). Defaults to an `empty Dictionary`. ### PerfdataWriter diff --git a/doc/14-features.md b/doc/14-features.md index 165a45319..d3fe25165 100644 --- a/doc/14-features.md +++ b/doc/14-features.md @@ -594,20 +594,33 @@ You can enable the feature using By default the `OpenTsdbWriter` object expects the TSD to listen at `127.0.0.1` on port `4242`. -The current naming schema is +The current default naming schema is: ``` -icinga.host. -icinga.service.. +icinga.host. +icinga.service.. ``` -for host and service checks. The tag host is always applied. +for host and service checks. The tag `host` is always applied. + +Icinga also sends perfdata warning, critical, minimum and maximum threshold values to OpenTSDB. +These are stored as new OpenTSDB metric names appended with `_warn`, `_crit`, `_min`, `_max`. +Values are only stored when the corresponding threshold exists in Icinga's perfdata. + +Example: +``` +icinga.service.. +icinga.service..._warn +icinga.service..._crit +icinga.service..._min +icinga.service..._max +``` To make sure Icinga 2 writes a valid metric into OpenTSDB some characters are replaced with `_` in the target name: ``` -\ (and space) +\ : (and space) ``` The resulting name in OpenTSDB might look like: @@ -652,6 +665,77 @@ with the following tags > You might want to set the tsd.core.auto_create_metrics setting to `true` > in your opentsdb.conf configuration file. +#### OpenTSDB Metric Prefix +Functionality exists to modify the built in OpenTSDB metric names that the plugin +writes to. By default this is `icinga.host` and `icinga.service.`. + +These prefixes can be modified as necessary to any arbitary string. The prefix +configuration also supports Icinga macros, so if you rather use `` +or any other variable instead of `` you may do so. + +To configure OpenTSDB metric name prefixes, create or modify the `host_template` and/or +`service_template` blocks in the `opentsdb.conf` file, to add a `metric` definition. +These modifications go hand in hand with the **OpenTSDB Custom Tag Support** detailed below, +and more information around macro use can be found there. + +Additionally, using custom Metric Prefixes or your own macros in the prefix may be +helpful if you are using the **OpenTSDB Generic Metric** functionality detailed below. + +An example configuration which includes prefix name modification: + +``` +object OpenTsdbWriter "opentsdb" { + host = "127.0.0.1" + port = 4242 + host_template = { + metric = "icinga.myhost" + tags = { + location = "$host.vars.location$" + checkcommand = "$host.check_command$" + } + } + service_template = { + metric = "icinga.service.$service.check_command$" + } +} +``` + +The above configuration will output the following naming schema: +``` +icinga.myhost. +icinga.service.. +``` +Note how `` is always appended in the default naming schema mode. + +#### OpenTSDB Generic Metric Naming Schema + +An alternate naming schema (`Generic Metrics`) is available where OpenTSDB metric names are more generic +and do not include the Icinga perfdata label in the metric name. Instead, +perfdata labels are stored in a tag `label` which is stored along with each perfdata value. + +This ultimately reduces the number of unique OpenTSDB metric names which may make +querying aggregate data easier. This also allows you to store all perfdata values for a +particular check inside one OpenTSDB metric name for each check. + +This alternate naming schema can be enabled by setting the following in the OpenTSDBWriter config: +`enable_generic_metrics = true` + +> **Tip** +> Consider using `Generic Metrics` along with the **OpenTSDB Metric Prefix** naming options +> described above + +An example of this naming schema when compared to the default is: + +``` +icinga.host +icinga.service. +``` + +> **Note** +> Note how `` does not appear in the OpenTSDB metric name +> when using `Generic Metrics`. Instead, a new tag `label` appears on each value written +> to OpenTSDB which contains the perfdata label. + #### Custom Tags In addition to the default tags listed above, it is possible to send diff --git a/etc/icinga2/features-available/opentsdb.conf b/etc/icinga2/features-available/opentsdb.conf index a98141daf..471a62268 100644 --- a/etc/icinga2/features-available/opentsdb.conf +++ b/etc/icinga2/features-available/opentsdb.conf @@ -6,17 +6,20 @@ object OpenTsdbWriter "opentsdb" { //host = "127.0.0.1" //port = 4242 + //enable_generic_metrics = false // Custom Tagging, refer to Icinga object type documentation for // OpenTsdbWriter //host_template = { + // metric = "icinga.host" // tags = { - // checkcommand = "$host.check_command$" + // zone = "$host.zone$" // } //} //service_template = { + // metric = "icinga.service.$service.check_command$" // tags = { - // checkcommand = "$service.check_command$" + // zone = "$service.zone$" // } //} } diff --git a/lib/perfdata/opentsdbwriter.cpp b/lib/perfdata/opentsdbwriter.cpp index 5b90ed52c..125ecaaed 100644 --- a/lib/perfdata/opentsdbwriter.cpp +++ b/lib/perfdata/opentsdbwriter.cpp @@ -155,6 +155,7 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C Host::Ptr host; Dictionary::Ptr config_tmpl; Dictionary::Ptr config_tmpl_tags; + String config_tmpl_metric; if (service) { host = service->GetHost(); @@ -168,13 +169,14 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C // Get the tags nested dictionary in the service/host template in the config if (config_tmpl) { config_tmpl_tags = config_tmpl->Get("tags"); + config_tmpl_metric = config_tmpl->Get("metric"); } String metric; std::map tags; // Resolve macros in configuration template and build custom tag list - if (config_tmpl_tags) { + if (config_tmpl_tags || !config_tmpl_metric.IsEmpty()) { // Configure config template macro resolver MacroProcessor::ResolverList resolvers; @@ -183,24 +185,46 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C resolvers.emplace_back("host", host); resolvers.emplace_back("icinga", IcingaApplication::GetInstance()); - ObjectLock olock(config_tmpl_tags); + // Resolve macros for the service and host template config line + if (config_tmpl_tags) { + ObjectLock olock(config_tmpl_tags); + + for (const Dictionary::Pair& pair : config_tmpl_tags) { + + String missing_macro; + Value value = MacroProcessor::ResolveMacros(pair.second, resolvers, cr, &missing_macro); + + if (!missing_macro.IsEmpty()) { + Log(LogDebug, "OpenTsdbWriter") + << "Unable to resolve macro:'" << missing_macro + << "' for this host or service."; + + continue; + } + + String tagname = Convert::ToString(pair.first); + tags[tagname] = EscapeTag(value); + + } + } - for (const Dictionary::Pair& pair : config_tmpl_tags) { + // Resolve macros for the metric config line + if (!config_tmpl_metric.IsEmpty()) { String missing_macro; - Value value = MacroProcessor::ResolveMacros(pair.second, resolvers, cr, &missing_macro); + Value value = MacroProcessor::ResolveMacros(config_tmpl_metric, resolvers, cr, &missing_macro); if (!missing_macro.IsEmpty()) { Log(LogDebug, "OpenTsdbWriter") << "Unable to resolve macro:'" << missing_macro << "' for this host or service."; - continue; } + else { - String tagname = Convert::ToString(pair.first); - tags[tagname] = EscapeTag(value); + config_tmpl_metric = Convert::ToString(value); + } } } @@ -210,13 +234,23 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C double ts = cr->GetExecutionEnd(); if (service) { - String serviceName = service->GetShortName(); - String escaped_serviceName = EscapeMetric(serviceName); - metric = "icinga.service." + escaped_serviceName; + if (!config_tmpl_metric.IsEmpty()) { + metric = config_tmpl_metric; + } else { + String serviceName = service->GetShortName(); + String escaped_serviceName = EscapeMetric(serviceName); + metric = "icinga.service." + escaped_serviceName; + } + SendMetric(checkable, metric + ".state", tags, service->GetState(), ts); + } else { - metric = "icinga.host"; + if (!config_tmpl_metric.IsEmpty()) { + metric = config_tmpl_metric; + } else { + metric = "icinga.host"; + } SendMetric(checkable, metric + ".state", tags, host->GetState(), ts); } @@ -280,20 +314,32 @@ void OpenTsdbWriter::SendPerfdata(const Checkable::Ptr& checkable, const String& continue; } } + + String metric_name; + std::map tags_new = tags; - String escaped_key = EscapeMetric(pdv->GetLabel()); - boost::algorithm::replace_all(escaped_key, "::", "."); + // Do not break original functionality where perfdata labels form + // part of the metric name + if (!GetEnableGenericMetrics()) { + String escaped_key = EscapeMetric(pdv->GetLabel()); + boost::algorithm::replace_all(escaped_key, "::", "."); + metric_name = metric + "." + escaped_key; + } else { + String escaped_key = EscapeTag(pdv->GetLabel()); + metric_name = metric; + tags_new["label"] = escaped_key; + } - SendMetric(checkable, metric + "." + escaped_key, tags, pdv->GetValue(), ts); + SendMetric(checkable, metric_name, tags_new, pdv->GetValue(), ts); if (pdv->GetCrit()) - SendMetric(checkable, metric + "." + escaped_key + "_crit", tags, pdv->GetCrit(), ts); + SendMetric(checkable, metric_name + "_crit", tags_new, pdv->GetCrit(), ts); if (pdv->GetWarn()) - SendMetric(checkable, metric + "." + escaped_key + "_warn", tags, pdv->GetWarn(), ts); + SendMetric(checkable, metric_name + "_warn", tags_new, pdv->GetWarn(), ts); if (pdv->GetMin()) - SendMetric(checkable, metric + "." + escaped_key + "_min", tags, pdv->GetMin(), ts); + SendMetric(checkable, metric_name + "_min", tags_new, pdv->GetMin(), ts); if (pdv->GetMax()) - SendMetric(checkable, metric + "." + escaped_key + "_max", tags, pdv->GetMax(), ts); + SendMetric(checkable, metric_name + "_max", tags_new, pdv->GetMax(), ts); } } @@ -360,6 +406,7 @@ String OpenTsdbWriter::EscapeTag(const String& str) boost::replace_all(result, " ", "_"); boost::replace_all(result, "\\", "_"); + boost::replace_all(result, ":", "_"); return result; } @@ -427,6 +474,10 @@ void OpenTsdbWriter::ValidateHostTemplate(const Lazy& lvalue, c { ObjectImpl::ValidateHostTemplate(lvalue, utils); + String metric = lvalue()->Get("metric"); + if (!MacroProcessor::ValidateMacroString(metric)) + BOOST_THROW_EXCEPTION(ValidationError(this, { "host_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'.")); + Dictionary::Ptr tags = lvalue()->Get("tags"); if (tags) { ObjectLock olock(tags); @@ -448,6 +499,10 @@ void OpenTsdbWriter::ValidateServiceTemplate(const Lazy& lvalue { ObjectImpl::ValidateServiceTemplate(lvalue, utils); + String metric = lvalue()->Get("metric"); + if (!MacroProcessor::ValidateMacroString(metric)) + BOOST_THROW_EXCEPTION(ValidationError(this, { "service_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'.")); + Dictionary::Ptr tags = lvalue()->Get("tags"); if (tags) { ObjectLock olock(tags); diff --git a/lib/perfdata/opentsdbwriter.ti b/lib/perfdata/opentsdbwriter.ti index 7a0219f01..626350a80 100644 --- a/lib/perfdata/opentsdbwriter.ti +++ b/lib/perfdata/opentsdbwriter.ti @@ -27,6 +27,9 @@ class OpenTsdbWriter : ConfigObject [config] Dictionary::Ptr service_template { default {{{ return new Dictionary(); }}} }; + [config] bool enable_generic_metrics { + default {{{ return false; }}} + }; [no_user_modify] bool connected; [no_user_modify] bool should_connect { @@ -36,11 +39,13 @@ class OpenTsdbWriter : ConfigObject validator OpenTsdbWriter { Dictionary host_template { + String metric; Dictionary "tags" { String "*"; }; }; Dictionary service_template { + String metric; Dictionary "tags" { String "*"; };