diff --git a/doc/09-object-types.md b/doc/09-object-types.md
index f0d208030..9c8a73baf 100644
--- a/doc/09-object-types.md
+++ b/doc/09-object-types.md
@@ -1703,8 +1703,9 @@ Configuration Attributes:
host | String | **Optional.** OpenTSDB host address. Defaults to `127.0.0.1`.
port | Number | **Optional.** OpenTSDB port. Defaults to `4242`.
enable\_ha | Boolean | **Optional.** Enable the high availability functionality. Only valid in a [cluster setup](06-distributed-monitoring.md#distributed-monitoring-high-availability-features). Defaults to `false`.
- host_template | Dictionary | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
- service_template | Dictionary | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
+ enable_generic_metrics | Boolean | **Optional.** Re-use metric names to store different perfdata values for a particular check. Use tags to distinguish perfdata instead of metric name. Defaults to `false`.
+ host_template | Dictionary | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
+ service_template | Dictionary | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). Defaults to an `empty Dictionary`.
### PerfdataWriter
diff --git a/doc/14-features.md b/doc/14-features.md
index 165a45319..d3fe25165 100644
--- a/doc/14-features.md
+++ b/doc/14-features.md
@@ -594,20 +594,33 @@ You can enable the feature using
By default the `OpenTsdbWriter` object expects the TSD to listen at
`127.0.0.1` on port `4242`.
-The current naming schema is
+The current default naming schema is:
```
-icinga.host.
-icinga.service..
+icinga.host.
+icinga.service..
```
-for host and service checks. The tag host is always applied.
+for host and service checks. The tag `host` is always applied.
+
+Icinga also sends perfdata warning, critical, minimum and maximum threshold values to OpenTSDB.
+These are stored as new OpenTSDB metric names appended with `_warn`, `_crit`, `_min`, `_max`.
+Values are only stored when the corresponding threshold exists in Icinga's perfdata.
+
+Example:
+```
+icinga.service..
+icinga.service..._warn
+icinga.service..._crit
+icinga.service..._min
+icinga.service..._max
+```
To make sure Icinga 2 writes a valid metric into OpenTSDB some characters are replaced
with `_` in the target name:
```
-\ (and space)
+\ : (and space)
```
The resulting name in OpenTSDB might look like:
@@ -652,6 +665,77 @@ with the following tags
> You might want to set the tsd.core.auto_create_metrics setting to `true`
> in your opentsdb.conf configuration file.
+#### OpenTSDB Metric Prefix
+Functionality exists to modify the built in OpenTSDB metric names that the plugin
+writes to. By default this is `icinga.host` and `icinga.service.`.
+
+These prefixes can be modified as necessary to any arbitary string. The prefix
+configuration also supports Icinga macros, so if you rather use ``
+or any other variable instead of `` you may do so.
+
+To configure OpenTSDB metric name prefixes, create or modify the `host_template` and/or
+`service_template` blocks in the `opentsdb.conf` file, to add a `metric` definition.
+These modifications go hand in hand with the **OpenTSDB Custom Tag Support** detailed below,
+and more information around macro use can be found there.
+
+Additionally, using custom Metric Prefixes or your own macros in the prefix may be
+helpful if you are using the **OpenTSDB Generic Metric** functionality detailed below.
+
+An example configuration which includes prefix name modification:
+
+```
+object OpenTsdbWriter "opentsdb" {
+ host = "127.0.0.1"
+ port = 4242
+ host_template = {
+ metric = "icinga.myhost"
+ tags = {
+ location = "$host.vars.location$"
+ checkcommand = "$host.check_command$"
+ }
+ }
+ service_template = {
+ metric = "icinga.service.$service.check_command$"
+ }
+}
+```
+
+The above configuration will output the following naming schema:
+```
+icinga.myhost.
+icinga.service..
+```
+Note how `` is always appended in the default naming schema mode.
+
+#### OpenTSDB Generic Metric Naming Schema
+
+An alternate naming schema (`Generic Metrics`) is available where OpenTSDB metric names are more generic
+and do not include the Icinga perfdata label in the metric name. Instead,
+perfdata labels are stored in a tag `label` which is stored along with each perfdata value.
+
+This ultimately reduces the number of unique OpenTSDB metric names which may make
+querying aggregate data easier. This also allows you to store all perfdata values for a
+particular check inside one OpenTSDB metric name for each check.
+
+This alternate naming schema can be enabled by setting the following in the OpenTSDBWriter config:
+`enable_generic_metrics = true`
+
+> **Tip**
+> Consider using `Generic Metrics` along with the **OpenTSDB Metric Prefix** naming options
+> described above
+
+An example of this naming schema when compared to the default is:
+
+```
+icinga.host
+icinga.service.
+```
+
+> **Note**
+> Note how `` does not appear in the OpenTSDB metric name
+> when using `Generic Metrics`. Instead, a new tag `label` appears on each value written
+> to OpenTSDB which contains the perfdata label.
+
#### Custom Tags
In addition to the default tags listed above, it is possible to send
diff --git a/etc/icinga2/features-available/opentsdb.conf b/etc/icinga2/features-available/opentsdb.conf
index a98141daf..471a62268 100644
--- a/etc/icinga2/features-available/opentsdb.conf
+++ b/etc/icinga2/features-available/opentsdb.conf
@@ -6,17 +6,20 @@
object OpenTsdbWriter "opentsdb" {
//host = "127.0.0.1"
//port = 4242
+ //enable_generic_metrics = false
// Custom Tagging, refer to Icinga object type documentation for
// OpenTsdbWriter
//host_template = {
+ // metric = "icinga.host"
// tags = {
- // checkcommand = "$host.check_command$"
+ // zone = "$host.zone$"
// }
//}
//service_template = {
+ // metric = "icinga.service.$service.check_command$"
// tags = {
- // checkcommand = "$service.check_command$"
+ // zone = "$service.zone$"
// }
//}
}
diff --git a/lib/perfdata/opentsdbwriter.cpp b/lib/perfdata/opentsdbwriter.cpp
index 5b90ed52c..125ecaaed 100644
--- a/lib/perfdata/opentsdbwriter.cpp
+++ b/lib/perfdata/opentsdbwriter.cpp
@@ -155,6 +155,7 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
Host::Ptr host;
Dictionary::Ptr config_tmpl;
Dictionary::Ptr config_tmpl_tags;
+ String config_tmpl_metric;
if (service) {
host = service->GetHost();
@@ -168,13 +169,14 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
// Get the tags nested dictionary in the service/host template in the config
if (config_tmpl) {
config_tmpl_tags = config_tmpl->Get("tags");
+ config_tmpl_metric = config_tmpl->Get("metric");
}
String metric;
std::map tags;
// Resolve macros in configuration template and build custom tag list
- if (config_tmpl_tags) {
+ if (config_tmpl_tags || !config_tmpl_metric.IsEmpty()) {
// Configure config template macro resolver
MacroProcessor::ResolverList resolvers;
@@ -183,24 +185,46 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
resolvers.emplace_back("host", host);
resolvers.emplace_back("icinga", IcingaApplication::GetInstance());
- ObjectLock olock(config_tmpl_tags);
+ // Resolve macros for the service and host template config line
+ if (config_tmpl_tags) {
+ ObjectLock olock(config_tmpl_tags);
+
+ for (const Dictionary::Pair& pair : config_tmpl_tags) {
+
+ String missing_macro;
+ Value value = MacroProcessor::ResolveMacros(pair.second, resolvers, cr, &missing_macro);
+
+ if (!missing_macro.IsEmpty()) {
+ Log(LogDebug, "OpenTsdbWriter")
+ << "Unable to resolve macro:'" << missing_macro
+ << "' for this host or service.";
+
+ continue;
+ }
+
+ String tagname = Convert::ToString(pair.first);
+ tags[tagname] = EscapeTag(value);
+
+ }
+ }
- for (const Dictionary::Pair& pair : config_tmpl_tags) {
+ // Resolve macros for the metric config line
+ if (!config_tmpl_metric.IsEmpty()) {
String missing_macro;
- Value value = MacroProcessor::ResolveMacros(pair.second, resolvers, cr, &missing_macro);
+ Value value = MacroProcessor::ResolveMacros(config_tmpl_metric, resolvers, cr, &missing_macro);
if (!missing_macro.IsEmpty()) {
Log(LogDebug, "OpenTsdbWriter")
<< "Unable to resolve macro:'" << missing_macro
<< "' for this host or service.";
- continue;
}
+ else {
- String tagname = Convert::ToString(pair.first);
- tags[tagname] = EscapeTag(value);
+ config_tmpl_metric = Convert::ToString(value);
+ }
}
}
@@ -210,13 +234,23 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
double ts = cr->GetExecutionEnd();
if (service) {
- String serviceName = service->GetShortName();
- String escaped_serviceName = EscapeMetric(serviceName);
- metric = "icinga.service." + escaped_serviceName;
+ if (!config_tmpl_metric.IsEmpty()) {
+ metric = config_tmpl_metric;
+ } else {
+ String serviceName = service->GetShortName();
+ String escaped_serviceName = EscapeMetric(serviceName);
+ metric = "icinga.service." + escaped_serviceName;
+ }
+
SendMetric(checkable, metric + ".state", tags, service->GetState(), ts);
+
} else {
- metric = "icinga.host";
+ if (!config_tmpl_metric.IsEmpty()) {
+ metric = config_tmpl_metric;
+ } else {
+ metric = "icinga.host";
+ }
SendMetric(checkable, metric + ".state", tags, host->GetState(), ts);
}
@@ -280,20 +314,32 @@ void OpenTsdbWriter::SendPerfdata(const Checkable::Ptr& checkable, const String&
continue;
}
}
+
+ String metric_name;
+ std::map tags_new = tags;
- String escaped_key = EscapeMetric(pdv->GetLabel());
- boost::algorithm::replace_all(escaped_key, "::", ".");
+ // Do not break original functionality where perfdata labels form
+ // part of the metric name
+ if (!GetEnableGenericMetrics()) {
+ String escaped_key = EscapeMetric(pdv->GetLabel());
+ boost::algorithm::replace_all(escaped_key, "::", ".");
+ metric_name = metric + "." + escaped_key;
+ } else {
+ String escaped_key = EscapeTag(pdv->GetLabel());
+ metric_name = metric;
+ tags_new["label"] = escaped_key;
+ }
- SendMetric(checkable, metric + "." + escaped_key, tags, pdv->GetValue(), ts);
+ SendMetric(checkable, metric_name, tags_new, pdv->GetValue(), ts);
if (pdv->GetCrit())
- SendMetric(checkable, metric + "." + escaped_key + "_crit", tags, pdv->GetCrit(), ts);
+ SendMetric(checkable, metric_name + "_crit", tags_new, pdv->GetCrit(), ts);
if (pdv->GetWarn())
- SendMetric(checkable, metric + "." + escaped_key + "_warn", tags, pdv->GetWarn(), ts);
+ SendMetric(checkable, metric_name + "_warn", tags_new, pdv->GetWarn(), ts);
if (pdv->GetMin())
- SendMetric(checkable, metric + "." + escaped_key + "_min", tags, pdv->GetMin(), ts);
+ SendMetric(checkable, metric_name + "_min", tags_new, pdv->GetMin(), ts);
if (pdv->GetMax())
- SendMetric(checkable, metric + "." + escaped_key + "_max", tags, pdv->GetMax(), ts);
+ SendMetric(checkable, metric_name + "_max", tags_new, pdv->GetMax(), ts);
}
}
@@ -360,6 +406,7 @@ String OpenTsdbWriter::EscapeTag(const String& str)
boost::replace_all(result, " ", "_");
boost::replace_all(result, "\\", "_");
+ boost::replace_all(result, ":", "_");
return result;
}
@@ -427,6 +474,10 @@ void OpenTsdbWriter::ValidateHostTemplate(const Lazy& lvalue, c
{
ObjectImpl::ValidateHostTemplate(lvalue, utils);
+ String metric = lvalue()->Get("metric");
+ if (!MacroProcessor::ValidateMacroString(metric))
+ BOOST_THROW_EXCEPTION(ValidationError(this, { "host_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'."));
+
Dictionary::Ptr tags = lvalue()->Get("tags");
if (tags) {
ObjectLock olock(tags);
@@ -448,6 +499,10 @@ void OpenTsdbWriter::ValidateServiceTemplate(const Lazy& lvalue
{
ObjectImpl::ValidateServiceTemplate(lvalue, utils);
+ String metric = lvalue()->Get("metric");
+ if (!MacroProcessor::ValidateMacroString(metric))
+ BOOST_THROW_EXCEPTION(ValidationError(this, { "service_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'."));
+
Dictionary::Ptr tags = lvalue()->Get("tags");
if (tags) {
ObjectLock olock(tags);
diff --git a/lib/perfdata/opentsdbwriter.ti b/lib/perfdata/opentsdbwriter.ti
index 7a0219f01..626350a80 100644
--- a/lib/perfdata/opentsdbwriter.ti
+++ b/lib/perfdata/opentsdbwriter.ti
@@ -27,6 +27,9 @@ class OpenTsdbWriter : ConfigObject
[config] Dictionary::Ptr service_template {
default {{{ return new Dictionary(); }}}
};
+ [config] bool enable_generic_metrics {
+ default {{{ return false; }}}
+ };
[no_user_modify] bool connected;
[no_user_modify] bool should_connect {
@@ -36,11 +39,13 @@ class OpenTsdbWriter : ConfigObject
validator OpenTsdbWriter {
Dictionary host_template {
+ String metric;
Dictionary "tags" {
String "*";
};
};
Dictionary service_template {
+ String metric;
Dictionary "tags" {
String "*";
};