Query prometheus counter across multiple instances

Question

I got several instances exposing a Prometheus counter and would like to aggregate all values over a certain period of time. I've been trying a lot of different things but can't get it working.

Let's assume my metric name is request_total. This metric has facets for path and status_code. My goal is to get an overall sum of this counter, without filtering it by any of its facets. If I run sum by (instance) (request_total), I get the following graph from Prometheus:

As we can see my counter seems to be correct for each instance. However, if I try to sum all those values with sum (request_total), I get the following result:

I'm pretty new to Prometheus but was expecting that the counter would actually not be reset and better cumulative. Could you please help me and tell me what I am missing here ?

Thanks in advance

Answer 1

Yes sum(request_total) should work and show the result across all the instances, and according to your graphs that's exactly what it does:

until ~8:30am there are two instances that report 4 and 11 requests, total of 15 which you can see in the second graph.

from ~8:33am to 8:42am only one instance reports one request and then another instance starts reporting one request as well which shows as going from 1 to 2 on the second accumulative graph.