Generating range vectors from return values in Prometheus queries

Question

I have a metric varnish_main_client_req of type counter and I want to set up an alert that triggers if the rate of requests drops/raises by a certain amount in a given time (e.g. "Amount of requests deviated in the last 2 min!").

Using the deriv() function should work much better than comparing relative values, but it can only be used with gauges. Is it possible to convert an ever increasing metric aka. counter to a rated metric aka. gauge?

Query: deriv(rate(varnish_main_client_req[2m])[5m])

Expectation: Prometheus calculates the rate of client requests over the last 2 mins and returns a derivative of the resulting values over the last 5 mins.

Actual result:

"error": "parse error at char 48: range specification must be preceded by a metric selector, but follows a *promql.Call instead"

Recording rules might be an option but it feels like a cheap workaround for something that should work with queries:

my_gauge_metric = rate(some_counter_metric[2m])

Answer 1

Solution

It's possible with the subquery-syntax (introduced in Prometheus version 2.7):

deriv(rate(varnish_main_client_req[2m])[5m:10s])

Warning: These subqueries are expensive, i.e. create very high load on Prometheus. Use recording-rules when you use these queries regularly (in alerts, etc.).

Subquery syntax

<instant_query>[<range>:<resolution>]

• instant_query: a PromQL-function which returns an instant-vector)
• range: offset (back in time) to start the first subquery
• resolution: the size of each of the subqueries.

It returns a range-vector.

In the example above, Prometheus runs rate() (= instant_query) 30 times (the first from 5 minutes ago to -4:50, ..., the last -0:10 to now). The resulting range-vector is input to the deriv()-function.

Another example (mostly available on all Prometheus instances):

deriv(rate(prometheus_http_request_duration_seconds_sum{job="prometheus"}[1m])[5m:10s])

Without the subquery-range ([5m:10s]), you'll get this error-message:

parse error at char 80: expected type range vector in call to function "deriv", got instant vector

Answer 2

Yes, you need to use a recording rule for this.

Prometheus calculates the rate of client requests over the last 2 mins and returns a derivative of the resulting values over the last 5 mins.

Herein lies the problem - at what interval should Prometheus synthesise this data?