Prometheus returns "ranges only allowed for vector selectors"

Question

Here's my query, which is supposed to show the delta for each counter of the form taskcnt.*:

delta(label_replace({__name__=~"taskcnt.*"}, "old_name", "$1", "__name__", "(.+)")[1w])

I'm getting:

Error executing query: 1:83: parse error: ranges only allowed for vector selectors

Basically, without the label_replace I'll get:

vector cannot contain metrics with the same labelset

How can I make this query work?

Answer 1

The Subquery is what you need indeed (credit to commenter above M. Doubez). This should work for you - it calculates the weekly delta by calculating the subquery each day (see [1w:1d])

delta(label_replace({__name__=~"desired_metric_prefix_.+_suffix"}, "metric_name", "$1", "__name__", "desired_metric_prefix_(.+)_suffix")[1w:1d])

Be sure that all of the metrics that match your regex are compatible with the label_replace and with the delta functions. If you're displaying this in Grafana, use Legend expression {{ metric_name }} to display the extracted metric_name for each series.

Answer 2

The {__name__=~"taskcnt.*"} may return multiple time series with identical set of labels, but with different metric names. For example, it may return the following time series:

taskcnt_foo{job="x",instance="y"}
taskcnt_bar{job="x",instance="y"}

As you can see, these time series have identical set of labels - {job="x",instance="y"}, while they have different metric names: taskcnt_foo vs taskcnt_bar.

The delta() function strips metric names from results, so it strips taskcnt_foo and taskcnt_bar names from the original time series and returns two time series with identical labelset - {job="x",instance="y"} when {__name__=~"taskcnt.*"} series selector is passed to it. This results in vector cannot contain metrics with the same labelset error, since the returned time series cannot be differentiated from each other.

Unfortunately Prometheus doesn't allow to use label_replace() or any other function inside delta()-like functions, which accept e.g. range vector - a series selector ending with square brackets (for example, {__name__=~"taskcnt.*"}[1w]).

There is a half-working solution by using subqueries, which is recommended in this answer. Unfortunately this solution has the following drawbacks:

• Prometheus subqueries may take significant amounts of additional CPU, memory and disk IO during execution, especially if the interval value is too small compared to the lookbehind window in square brackets: [window:interval].
• Prometheus subqueries may miss original raw data during calculations when the interval in square brackets is too big.
• Prometheus subqueries are easy to mis-use if their behavior isn't understood clearly.

These issues are addressed by VictoriaMetrics in MetricsQL with keep_metric_names modifier, which can be applied to any function. For example, the following MetricsQL query doesn't strip metric names from results, so it doesn't return neither vector cannot contain metrics with the same labelset nor ranges only allowed for vector selectors errors:

increase({__name__=~"taskcnt.*"}[1w]) keep_metric_names