Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can we add extra label to Prometheus metrics?

Suppose we are collecting the same metrics for one month and now we want to modify the metrics to have extra label (in the old data as well), how can we do that. Existing metric:

mongodb_exporter_last_scrape_duration_seconds{instance="127.0.0.1:9216",job="mongo"}

Want to change that to:

mongodb_exporter_last_scrape_duration_seconds{cluster="stage", instance="127.0.0.1:9216",job="mongo"}   
like image 310
Nipun Talukdar Avatar asked Dec 04 '17 07:12

Nipun Talukdar


People also ask

How do you add labels to the metric system?

If you want to add a label unconditionally to every metric returned from a specific scrape job, then just add the label in the service discovery (e.g. "file_sd_configs" or "static_configs"). Different labels can be applied for different groups of targets, e.g.

How do I change the label value in Prometheus?

A common use case for relabeling is to set or overwrite the value of a label. This can be done using the replace action, which is the default if the action field is not specified.

What is label in Prometheus metrics?

A label is a certain attribute of a metric. Generally, labels are populated by metric producers (servers in the example above). However, in Prometheus, it's possible to enrich a metric with some static labels based on the producer's identity while recording it on the Prometheus node's side.


1 Answers

Prometheus doesn't provide the ability to add new labels to historical time series. But it is possible to add missing labels during queries with label_replace() function:

label_replace(
  mongodb_exporter_last_scrape_duration_seconds,
  "cluster",
  "staging",
  "cluster",
  ""
)

This label_replace() call adds cluster="staging" label only if the cluster label is missing in the original time series selected with mongodb_exporter_last_scrape_duration_seconds series selector.

like image 106
valyala Avatar answered Oct 06 '22 04:10

valyala