Prometheus Query Overall average under a time interval

Question

How can I find the overall average of metrics over time interval ?

avg(metric) = overall average value but

avg_over_time(metrics[interval]) = averages value per label

avg( avg_over_time(metric[scrape interval]) ) won't be same as(when the data is not continuous and denominator value is different) avg(metric) !!!!

Given a scenario, what will be the possible way to find the overall average over a time period.

Eg: Find the average response time now and Find the average response time(over all) of all the request triggered in last one hour.

The number will be helpful to notify a performance issue with latest upgrades.

Answer 1

You need to calculate the average a bit more manually:

    sum(sum_over_time(metric[interval]))
/
    sum(count_over_time(metric[interval]))

Note that this is for data in a gauge, you'd need a different approach for data from a counter or summary.