tyler.durden
/
icinga2
mirror of https://github.com/Icinga/icinga2.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Added OpenTSDB Metric prefix naming support 

Added OpenTSDB Generic Metric functionality
This commit is contained in:
Ant1x 2019-10-23 23:58:27 +11:00
parent 0b8fdfd7f2
commit 166e79fee1
5 changed files with 175 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
doc/09-object-types.md
View File

@ -1703,8 +1703,9 @@ Configuration Attributes:
host | String | **Optional.** OpenTSDB host address. Defaults to `127.0.0.1`.
port | Number | **Optional.** OpenTSDB port. Defaults to `4242`.
enable\_ha | Boolean | **Optional.** Enable the high availability functionality. Only valid in a [cluster setup](06-distributed-monitoring.md#distributed-monitoring-high-availability-features). Defaults to `false`.
host_template | Dictionary | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
service_template | Dictionary | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
enable_generic_metrics | Boolean | **Optional.** Re-use metric names to store different perfdata values for a particular check. Use tags to distinguish perfdata instead of metric name. Defaults to `false`.
host_template | Dictionary | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
service_template | Dictionary | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). Defaults to an `empty Dictionary`.
### PerfdataWriter <a id="objecttype-perfdatawriter"></a>

94
doc/14-features.md
View File

@ -594,20 +594,33 @@ You can enable the feature using
By default the `OpenTsdbWriter` object expects the TSD to listen at
`127.0.0.1` on port `4242`.
The current naming schema is
The current default naming schema is:
```
icinga.host.<metricname>
icinga.service.<servicename>.<metricname>
icinga.host.<perfdata_metric_label>
icinga.service.<servicename>.<perfdata_metric_label>
```
for host and service checks. The tag host is always applied.
for host and service checks. The tag `host` is always applied.
Icinga also sends perfdata warning, critical, minimum and maximum threshold values to OpenTSDB.
These are stored as new OpenTSDB metric names appended with `_warn`, `_crit`, `_min`, `_max`.
Values are only stored when the corresponding threshold exists in Icinga's perfdata.
Example:
```
icinga.service.<servicename>.<perfdata_metric_label>
icinga.service.<servicename>.<perfdata_metric_label>._warn
icinga.service.<servicename>.<perfdata_metric_label>._crit
icinga.service.<servicename>.<perfdata_metric_label>._min
icinga.service.<servicename>.<perfdata_metric_label>._max
```
To make sure Icinga 2 writes a valid metric into OpenTSDB some characters are replaced
with `_` in the target name:
```
\ (and space)
\ : (and space)
```
The resulting name in OpenTSDB might look like:
@ -652,6 +665,77 @@ with the following tags
> You might want to set the tsd.core.auto_create_metrics setting to `true`
> in your opentsdb.conf configuration file.
#### OpenTSDB Metric Prefix <a id="opentsdb-metric-prefix"></a>
Functionality exists to modify the built in OpenTSDB metric names that the plugin
writes to. By default this is `icinga.host` and `icinga.service.<servicename>`.
These prefixes can be modified as necessary to any arbitary string. The prefix
configuration also supports Icinga macros, so if you rather use `<checkcommand>`
or any other variable instead of `<servicename>` you may do so.
To configure OpenTSDB metric name prefixes, create or modify the `host_template` and/or
`service_template` blocks in the `opentsdb.conf` file, to add a `metric` definition.
These modifications go hand in hand with the **OpenTSDB Custom Tag Support** detailed below,
and more information around macro use can be found there.
Additionally, using custom Metric Prefixes or your own macros in the prefix may be
helpful if you are using the **OpenTSDB Generic Metric** functionality detailed below.
An example configuration which includes prefix name modification:
```
object OpenTsdbWriter "opentsdb" {
host = "127.0.0.1"
port = 4242
host_template = {
metric = "icinga.myhost"
tags = {
location = "$host.vars.location$"
checkcommand = "$host.check_command$"
}
}
service_template = {
metric = "icinga.service.$service.check_command$"
}
}
```
The above configuration will output the following naming schema:
```
icinga.myhost.<perfdata_metric_label>
icinga.service.<check_command_name>.<perfdata_metric_label>
```
Note how `<perfdata_metric_label>` is always appended in the default naming schema mode.
#### OpenTSDB Generic Metric Naming Schema <a id="opentsdb-generic-metrics"></a>
An alternate naming schema (`Generic Metrics`) is available where OpenTSDB metric names are more generic
and do not include the Icinga perfdata label in the metric name. Instead,
perfdata labels are stored in a tag `label` which is stored along with each perfdata value.
This ultimately reduces the number of unique OpenTSDB metric names which may make
querying aggregate data easier. This also allows you to store all perfdata values for a
particular check inside one OpenTSDB metric name for each check.
This alternate naming schema can be enabled by setting the following in the OpenTSDBWriter config:
`enable_generic_metrics = true`
> **Tip**
> Consider using `Generic Metrics` along with the **OpenTSDB Metric Prefix** naming options
> described above
An example of this naming schema when compared to the default is:
```
icinga.host
icinga.service.<servicename>
```
> **Note**
> Note how `<perfdata_metric_label>` does not appear in the OpenTSDB metric name
> when using `Generic Metrics`. Instead, a new tag `label` appears on each value written
> to OpenTSDB which contains the perfdata label.
#### Custom Tags <a id="opentsdb-custom-tags"></a>
In addition to the default tags listed above, it is possible to send

7
etc/icinga2/features-available/opentsdb.conf
View File

@ -6,17 +6,20 @@
object OpenTsdbWriter "opentsdb" {
//host = "127.0.0.1"
//port = 4242
//enable_generic_metrics = false
// Custom Tagging, refer to Icinga object type documentation for
// OpenTsdbWriter
//host_template = {
// metric = "icinga.host"
// tags = {
// checkcommand = "$host.check_command$"
// zone = "$host.zone$"
// }
//}
//service_template = {
// metric = "icinga.service.$service.check_command$"
// tags = {
// checkcommand = "$service.check_command$"
// zone = "$service.zone$"
// }
//}
}

67
lib/perfdata/opentsdbwriter.cpp
View File

@ -155,6 +155,7 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
Host::Ptr host;
Dictionary::Ptr config_tmpl;
Dictionary::Ptr config_tmpl_tags;
String config_tmpl_metric;
if (service) {
host = service->GetHost();
@ -168,13 +169,14 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
// Get the tags nested dictionary in the service/host template in the config
if (config_tmpl) {
config_tmpl_tags = config_tmpl->Get("tags");
config_tmpl_metric = config_tmpl->Get("metric");
}
String metric;
std::map<String, String> tags;
// Resolve macros in configuration template and build custom tag list
if (config_tmpl_tags) {
if (config_tmpl_tags || !config_tmpl_metric.IsEmpty()) {
// Configure config template macro resolver
MacroProcessor::ResolverList resolvers;
@ -183,6 +185,8 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
resolvers.emplace_back("host", host);
resolvers.emplace_back("icinga", IcingaApplication::GetInstance());
// Resolve macros for the service and host template config line
if (config_tmpl_tags) {
ObjectLock olock(config_tmpl_tags);
for (const Dictionary::Pair& pair : config_tmpl_tags) {
@ -204,19 +208,49 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
}
}
// Resolve macros for the metric config line
if (!config_tmpl_metric.IsEmpty()) {
String missing_macro;
Value value = MacroProcessor::ResolveMacros(config_tmpl_metric, resolvers, cr, &missing_macro);
if (!missing_macro.IsEmpty()) {
Log(LogDebug, "OpenTsdbWriter")
<< "Unable to resolve macro:'" << missing_macro
<< "' for this host or service.";
}
else {
config_tmpl_metric = Convert::ToString(value);
}
}
}
String escaped_hostName = EscapeTag(host->GetName());
tags["host"] = escaped_hostName;
double ts = cr->GetExecutionEnd();
if (service) {
if (!config_tmpl_metric.IsEmpty()) {
metric = config_tmpl_metric;
} else {
String serviceName = service->GetShortName();
String escaped_serviceName = EscapeMetric(serviceName);
metric = "icinga.service." + escaped_serviceName;
}
SendMetric(checkable, metric + ".state", tags, service->GetState(), ts);
} else {
if (!config_tmpl_metric.IsEmpty()) {
metric = config_tmpl_metric;
} else {
metric = "icinga.host";
}
SendMetric(checkable, metric + ".state", tags, host->GetState(), ts);
}
@ -281,19 +315,31 @@ void OpenTsdbWriter::SendPerfdata(const Checkable::Ptr& checkable, const String&
}
}
String metric_name;
std::map<String, String> tags_new = tags;
// Do not break original functionality where perfdata labels form
// part of the metric name
if (!GetEnableGenericMetrics()) {
String escaped_key = EscapeMetric(pdv->GetLabel());
boost::algorithm::replace_all(escaped_key, "::", ".");
metric_name = metric + "." + escaped_key;
} else {
String escaped_key = EscapeTag(pdv->GetLabel());
metric_name = metric;
tags_new["label"] = escaped_key;
}
SendMetric(checkable, metric + "." + escaped_key, tags, pdv->GetValue(), ts);
SendMetric(checkable, metric_name, tags_new, pdv->GetValue(), ts);
if (pdv->GetCrit())
SendMetric(checkable, metric + "." + escaped_key + "_crit", tags, pdv->GetCrit(), ts);
SendMetric(checkable, metric_name + "_crit", tags_new, pdv->GetCrit(), ts);
if (pdv->GetWarn())
SendMetric(checkable, metric + "." + escaped_key + "_warn", tags, pdv->GetWarn(), ts);
SendMetric(checkable, metric_name + "_warn", tags_new, pdv->GetWarn(), ts);
if (pdv->GetMin())
SendMetric(checkable, metric + "." + escaped_key + "_min", tags, pdv->GetMin(), ts);
SendMetric(checkable, metric_name + "_min", tags_new, pdv->GetMin(), ts);
if (pdv->GetMax())
SendMetric(checkable, metric + "." + escaped_key + "_max", tags, pdv->GetMax(), ts);
SendMetric(checkable, metric_name + "_max", tags_new, pdv->GetMax(), ts);
}
}
@ -360,6 +406,7 @@ String OpenTsdbWriter::EscapeTag(const String& str)
boost::replace_all(result, " ", "_");
boost::replace_all(result, "\\", "_");
boost::replace_all(result, ":", "_");
return result;
}
@ -427,6 +474,10 @@ void OpenTsdbWriter::ValidateHostTemplate(const Lazy<Dictionary::Ptr>& lvalue, c
{
ObjectImpl<OpenTsdbWriter>::ValidateHostTemplate(lvalue, utils);
String metric = lvalue()->Get("metric");
if (!MacroProcessor::ValidateMacroString(metric))
BOOST_THROW_EXCEPTION(ValidationError(this, { "host_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'."));
Dictionary::Ptr tags = lvalue()->Get("tags");
if (tags) {
ObjectLock olock(tags);
@ -448,6 +499,10 @@ void OpenTsdbWriter::ValidateServiceTemplate(const Lazy<Dictionary::Ptr>& lvalue
{
ObjectImpl<OpenTsdbWriter>::ValidateServiceTemplate(lvalue, utils);
String metric = lvalue()->Get("metric");
if (!MacroProcessor::ValidateMacroString(metric))
BOOST_THROW_EXCEPTION(ValidationError(this, { "service_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'."));
Dictionary::Ptr tags = lvalue()->Get("tags");
if (tags) {
ObjectLock olock(tags);

5
lib/perfdata/opentsdbwriter.ti
View File

@ -27,6 +27,9 @@ class OpenTsdbWriter : ConfigObject
[config] Dictionary::Ptr service_template {
default {{{ return new Dictionary(); }}}
};
[config] bool enable_generic_metrics {
default {{{ return false; }}}
};
[no_user_modify] bool connected;
[no_user_modify] bool should_connect {
@ -36,11 +39,13 @@ class OpenTsdbWriter : ConfigObject
validator OpenTsdbWriter {
Dictionary host_template {
String metric;
Dictionary "tags" {
String "*";
};
};
Dictionary service_template {
String metric;
Dictionary "tags" {
String "*";
};