prometheus rate on series by regex

Question

I am using the following query to get some metrics based on the name:

{__name__=~"bus_listener.+_processed"}

There are multiple metrics that match this name and multiple apps are publishing these metrics.

I am trying to calculate a rate on this, with:

rate({__name__=~"bus_listener.+_processed"}[5m])

But this gives me the following error:

vector cannot contain metrics with the same labelset

I cannot use recording metrics, I only have access to grafana, which reads metrics from prometheus.

How can I get this rate using a regex?

Answer 1

Sounds like you have multiple metrics with the same labels (except for __name__). rate() keeps all labels except __name__, but it drops __name__ to avoid any confusion. Meaning that if you have two time series like:

bus_listener_foo_processed{job="a_job"} 1
bus_listener_bar_processed{job="a_job"} 2

putting them through rate() will result in two time series both with the same labelset:

{job="a_job"} 0.1
{job="a_job"} 0.2

In theory you could duplicate the __name__ label as some other label by using label_replace() first and applying rate() on the result of that, resulting in different labelsets for each original timeseries. However, since you can only compute rate() directly on a timeseries (not the output of another function) you can only do this by using subqueries, which is both heavyweight and slower than it would otherwise be:

rate(label_replace({__name__=~"bus_listener.+_processed"}, "old_name", "$1", "__name__", "(.+)")[5m:1m])

(Optionally replacing 1m with something close to your scrape interval, so there's as little aliasing going on as possible.)

But ideally, if you do have access to the Prometheus configuration (which doesn't seem likely, since you say you can't use recording rules) you should use metric relabeling at ingestion time to extract the various bits of the metric name into separate labels, so you won't have to jump through hoops later on. Or have the service that exports the original metrics use labels instead of concatenating them into the metric name.