Added OpenTSDB Metric prefix naming support

Added OpenTSDB Generic Metric functionality

doc/09-object-types.md

@@ -1703,8 +1703,9 @@ Configuration Attributes:
   host            	    | String                | **Optional.** OpenTSDB host address. Defaults to `127.0.0.1`.
   port            	    | Number                | **Optional.** OpenTSDB port. Defaults to `4242`.
   enable\_ha                | Boolean               | **Optional.** Enable the high availability functionality. Only valid in a [cluster setup](06-distributed-monitoring.md#distributed-monitoring-high-availability-features). Defaults to `false`.
-  host_template             | Dictionary                | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
+  enable_generic_metrics    | Boolean               | **Optional.** Re-use metric names to store different perfdata values for a particular check. Use tags to distinguish perfdata instead of metric name. Defaults to `false`.
-  service_template          | Dictionary                | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
+  host_template             | Dictionary                | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
+  service_template          | Dictionary                | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). Defaults to an `empty Dictionary`.
 
 
 ### PerfdataWriter <a id="objecttype-perfdatawriter"></a>

doc/14-features.md

@@ -594,20 +594,33 @@ You can enable the feature using
 By default the `OpenTsdbWriter` object expects the TSD to listen at
 `127.0.0.1` on port `4242`.
 
-The current naming schema is
+The current default naming schema is:
 
 ```
-icinga.host.<metricname>
+icinga.host.<perfdata_metric_label>
-icinga.service.<servicename>.<metricname>
+icinga.service.<servicename>.<perfdata_metric_label>
 ```
 
-for host and service checks. The tag host is always applied.
+for host and service checks. The tag `host` is always applied.
+
+Icinga also sends perfdata warning, critical, minimum and maximum threshold values to OpenTSDB.
+These are stored as new OpenTSDB metric names appended with `_warn`, `_crit`, `_min`, `_max`.
+Values are only stored when the corresponding threshold exists in Icinga's perfdata.
+
+Example:
+```
+icinga.service.<servicename>.<perfdata_metric_label>
+icinga.service.<servicename>.<perfdata_metric_label>._warn
+icinga.service.<servicename>.<perfdata_metric_label>._crit
+icinga.service.<servicename>.<perfdata_metric_label>._min
+icinga.service.<servicename>.<perfdata_metric_label>._max
+```
 
 To make sure Icinga 2 writes a valid metric into OpenTSDB some characters are replaced
 with `_` in the target name:
 
 ```
-\  (and space)
+\ :  (and space)
 ```
 
 The resulting name in OpenTSDB might look like:
@@ -652,6 +665,77 @@ with the following tags
 > You might want to set the tsd.core.auto_create_metrics setting to `true`
 > in your opentsdb.conf configuration file.
 
+#### OpenTSDB Metric Prefix <a id="opentsdb-metric-prefix"></a>
+Functionality exists to modify the built in OpenTSDB metric names that the plugin
+writes to. By default this is `icinga.host` and `icinga.service.<servicename>`.
+
+These prefixes can be modified as necessary to any arbitary string. The prefix
+configuration also supports Icinga macros, so if you rather use `<checkcommand>` 
+or any other variable instead of `<servicename>` you may do so.
+
+To configure OpenTSDB metric name prefixes, create or modify the `host_template` and/or
+`service_template` blocks in the `opentsdb.conf` file, to add a `metric` definition.
+These modifications go hand in hand with the **OpenTSDB Custom Tag Support** detailed below,
+and more information around macro use can be found there.
+
+Additionally, using custom Metric Prefixes or your own macros in the prefix may be
+helpful if you are using the **OpenTSDB Generic Metric** functionality detailed below.
+
+An example configuration which includes prefix name modification:
+
+```
+object OpenTsdbWriter "opentsdb" {
+  host = "127.0.0.1"
+  port = 4242
+  host_template = {
+    metric = "icinga.myhost"
+    tags = {
+      location = "$host.vars.location$"
+      checkcommand = "$host.check_command$"
+    }
+  }
+  service_template = {
+    metric = "icinga.service.$service.check_command$"
+  }
+}
+```
+
+The above configuration will output the following naming schema:
+```
+icinga.myhost.<perfdata_metric_label>
+icinga.service.<check_command_name>.<perfdata_metric_label>
+```
+Note how `<perfdata_metric_label>` is always appended in the default naming schema mode.
+
+#### OpenTSDB Generic Metric Naming Schema <a id="opentsdb-generic-metrics"></a>
+
+An alternate naming schema (`Generic Metrics`) is available where OpenTSDB metric names are more generic
+and do not include the Icinga perfdata label in the metric name. Instead,
+perfdata labels are stored in a tag `label` which is stored along with each perfdata value.
+
+This ultimately reduces the number of unique OpenTSDB metric names which may make
+querying aggregate data easier. This also allows you to store all perfdata values for a
+particular check inside one OpenTSDB metric name for each check.
+
+This alternate naming schema can be enabled by setting the following in the OpenTSDBWriter config:
+`enable_generic_metrics = true`
+
+> **Tip**
+> Consider using `Generic Metrics` along with the **OpenTSDB Metric Prefix** naming options
+> described above
+
+An example of this naming schema when compared to the default is:
+
+```
+icinga.host
+icinga.service.<servicename>
+```
+
+> **Note**
+> Note how `<perfdata_metric_label>` does not appear in the OpenTSDB metric name
+> when using `Generic Metrics`. Instead, a new tag `label` appears on each value written
+> to OpenTSDB which contains the perfdata label.
+
 #### Custom Tags <a id="opentsdb-custom-tags"></a>
 
 In addition to the default tags listed above, it is possible to send

etc/icinga2/features-available/opentsdb.conf

@@ -6,17 +6,20 @@
 object OpenTsdbWriter "opentsdb" {
   //host = "127.0.0.1"
   //port = 4242
+  //enable_generic_metrics = false
 
   // Custom Tagging, refer to Icinga object type documentation for
   // OpenTsdbWriter
   //host_template = {
+  //  metric = "icinga.host"
   //  tags = {
-  //    checkcommand = "$host.check_command$"
+  //    zone = "$host.zone$"
   //  }
   //}
   //service_template = {
+  //  metric = "icinga.service.$service.check_command$"
   //  tags = {
-  //    checkcommand = "$service.check_command$"
+  //    zone = "$service.zone$"
   //  }
   //}
 }

lib/perfdata/opentsdbwriter.cpp

@@ -155,6 +155,7 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
 	Host::Ptr host;
 	Dictionary::Ptr config_tmpl;
 	Dictionary::Ptr config_tmpl_tags;
+	String config_tmpl_metric;
 
 	if (service) {
 		host = service->GetHost();
@@ -168,13 +169,14 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
 	// Get the tags nested dictionary in the service/host template in the config
 	if (config_tmpl) {
 		config_tmpl_tags = config_tmpl->Get("tags");
+		config_tmpl_metric = config_tmpl->Get("metric");
 	}
 
 	String metric;
 	std::map<String, String> tags;
 
 	// Resolve macros in configuration template and build custom tag list
-	if (config_tmpl_tags) {
+	if (config_tmpl_tags || !config_tmpl_metric.IsEmpty()) {
 
 		// Configure config template macro resolver
 		MacroProcessor::ResolverList resolvers;
@@ -183,24 +185,46 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
 		resolvers.emplace_back("host", host);
 		resolvers.emplace_back("icinga", IcingaApplication::GetInstance());
 		
-		ObjectLock olock(config_tmpl_tags);
+		// Resolve macros for the service and host template config line
+		if (config_tmpl_tags) {
+			ObjectLock olock(config_tmpl_tags);
+			
+			for (const Dictionary::Pair& pair : config_tmpl_tags) {
+				
+				String missing_macro;
+				Value value = MacroProcessor::ResolveMacros(pair.second, resolvers, cr, &missing_macro);
+				
+				if (!missing_macro.IsEmpty()) {
+					Log(LogDebug, "OpenTsdbWriter")
+						<< "Unable to resolve macro:'" << missing_macro 
+						<< "' for this host or service.";
+					
+					continue;
+				}
+				
+				String tagname = Convert::ToString(pair.first);
+				tags[tagname] = EscapeTag(value);
+				
+			}
+		}
 		
-		for (const Dictionary::Pair& pair : config_tmpl_tags) {
+		// Resolve macros for the metric config line
+		if (!config_tmpl_metric.IsEmpty()) {
 			
 			String missing_macro;
-			Value value = MacroProcessor::ResolveMacros(pair.second, resolvers, cr, &missing_macro);
+			Value value = MacroProcessor::ResolveMacros(config_tmpl_metric, resolvers, cr, &missing_macro);
 			
 			if (!missing_macro.IsEmpty()) {
 				Log(LogDebug, "OpenTsdbWriter")
 					<< "Unable to resolve macro:'" << missing_macro 
 					<< "' for this host or service.";
 				
-				continue;
 			}
+			else {
 			
-			String tagname = Convert::ToString(pair.first);
+				config_tmpl_metric = Convert::ToString(value);
-			tags[tagname] = EscapeTag(value);
 			
+			}
 		}
 	}
 
@@ -210,13 +234,23 @@ void OpenTsdbWriter::CheckResultHandler(const Checkable::Ptr& checkable, const C
 	double ts = cr->GetExecutionEnd();
 
 	if (service) {
-		String serviceName = service->GetShortName();
-		String escaped_serviceName = EscapeMetric(serviceName);
-		metric = "icinga.service." + escaped_serviceName;
 
+		if (!config_tmpl_metric.IsEmpty()) {
+			metric = config_tmpl_metric;
+		} else {
+			String serviceName = service->GetShortName();
+			String escaped_serviceName = EscapeMetric(serviceName);
+			metric = "icinga.service." + escaped_serviceName;
+		}
+		
 		SendMetric(checkable, metric + ".state", tags, service->GetState(), ts);
+
 	} else {
-		metric = "icinga.host";
+		if (!config_tmpl_metric.IsEmpty()) {
+			metric = config_tmpl_metric;
+		} else {
+			metric = "icinga.host";
+		}
 		SendMetric(checkable, metric + ".state", tags, host->GetState(), ts);
 	}
 
@@ -280,20 +314,32 @@ void OpenTsdbWriter::SendPerfdata(const Checkable::Ptr& checkable, const String&
 				continue;
 			}
 		}
+		
+		String metric_name;
+		std::map<String, String> tags_new = tags;
 
-		String escaped_key = EscapeMetric(pdv->GetLabel());
+		// Do not break original functionality where perfdata labels form
-		boost::algorithm::replace_all(escaped_key, "::", ".");
+		// part of the metric name
+		if (!GetEnableGenericMetrics()) {
+			String escaped_key = EscapeMetric(pdv->GetLabel());
+			boost::algorithm::replace_all(escaped_key, "::", ".");
+			metric_name = metric + "." + escaped_key;
+		} else {
+			String escaped_key = EscapeTag(pdv->GetLabel());
+			metric_name = metric;
+			tags_new["label"] = escaped_key;
+		}
 
-		SendMetric(checkable, metric + "." + escaped_key, tags, pdv->GetValue(), ts);
+		SendMetric(checkable, metric_name, tags_new, pdv->GetValue(), ts);
 
 		if (pdv->GetCrit())
-			SendMetric(checkable, metric + "." + escaped_key + "_crit", tags, pdv->GetCrit(), ts);
+			SendMetric(checkable, metric_name + "_crit", tags_new, pdv->GetCrit(), ts);
 		if (pdv->GetWarn())
-			SendMetric(checkable, metric + "." + escaped_key + "_warn", tags, pdv->GetWarn(), ts);
+			SendMetric(checkable, metric_name + "_warn", tags_new, pdv->GetWarn(), ts);
 		if (pdv->GetMin())
-			SendMetric(checkable, metric + "." + escaped_key + "_min", tags, pdv->GetMin(), ts);
+			SendMetric(checkable, metric_name + "_min", tags_new, pdv->GetMin(), ts);
 		if (pdv->GetMax())
-			SendMetric(checkable, metric + "." + escaped_key + "_max", tags, pdv->GetMax(), ts);
+			SendMetric(checkable, metric_name + "_max", tags_new, pdv->GetMax(), ts);
 	}
 }
 
@@ -360,6 +406,7 @@ String OpenTsdbWriter::EscapeTag(const String& str)
 
 	boost::replace_all(result, " ", "_");
 	boost::replace_all(result, "\\", "_");
+	boost::replace_all(result, ":", "_");
 
 	return result;
 }
@@ -427,6 +474,10 @@ void OpenTsdbWriter::ValidateHostTemplate(const Lazy<Dictionary::Ptr>& lvalue, c
 {
 	ObjectImpl<OpenTsdbWriter>::ValidateHostTemplate(lvalue, utils);
 
+	String metric = lvalue()->Get("metric");
+	if (!MacroProcessor::ValidateMacroString(metric))
+		BOOST_THROW_EXCEPTION(ValidationError(this, { "host_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'."));
+
 	Dictionary::Ptr tags = lvalue()->Get("tags");
 	if (tags) {
 		ObjectLock olock(tags);
@@ -448,6 +499,10 @@ void OpenTsdbWriter::ValidateServiceTemplate(const Lazy<Dictionary::Ptr>& lvalue
 {
 	ObjectImpl<OpenTsdbWriter>::ValidateServiceTemplate(lvalue, utils);
 
+	String metric = lvalue()->Get("metric");
+	if (!MacroProcessor::ValidateMacroString(metric))
+		BOOST_THROW_EXCEPTION(ValidationError(this, { "service_template", "metric" }, "Closing $ not found in macro format string '" + metric + "'."));
+
 	Dictionary::Ptr tags = lvalue()->Get("tags");
 	if (tags) {
 		ObjectLock olock(tags);

lib/perfdata/opentsdbwriter.ti

@@ -27,6 +27,9 @@ class OpenTsdbWriter : ConfigObject
 	[config] Dictionary::Ptr service_template {
 		default {{{ return new Dictionary(); }}}
 	};
+	[config] bool enable_generic_metrics {
+		default {{{ return false; }}}
+	};
 
 	[no_user_modify] bool connected;
 	[no_user_modify] bool should_connect {
@@ -36,11 +39,13 @@ class OpenTsdbWriter : ConfigObject
 
 validator OpenTsdbWriter {
 	Dictionary host_template {
+		String metric;
 		Dictionary "tags" {
 			String "*";
 		};
 	};
 	Dictionary service_template {
+		String metric;
 		Dictionary "tags" {
 			String "*";
 		};