Prometheus latency graph in histogram and calculate percentile

Question

I need to plot latency graph on prometheus by the histogram time-series, but I've been unsuccessful to display a histogram in grafana.

What I expect is to be able to show: Y-axis is latency, x-axis is timeseries.

Each line representing the p50,p75,p90,p100 - aggregated for a given time window. A sample metric would be the request time of an nginx.

suppose if i have a histogram like this,

nginx_request_time_bucket(le=1) 1,
nginx_request_time_bucket(le=10) 2,
nginx_request_time_bucket(le=60) 2,
nginx_request_time_bucket(le=+inf) 5

An example graph of what I am looking for is in this link, [][] [click]: https://www.instana.com/blog/how-to-measure-latency-properly-in-7-minutes/

I tried to picture histogram with heatmap using this query but this doesn't give me what im looking for. Im looking something similar to the graph

histogram_quantile(0.75, sum(rate(nginx_request_time_bucket[5m])) by (le))

Any help here is highly appreciated!

Answer 1

You need to set up a separate query on a Grafana graph per each needed percentile. For example, if you need p50, p75, p90 and p100 latencies over the last 5 minutes, then the following four separate queries should be set up in Grafana:

histogram_quantile(0.50, sum(rate(nginx_request_time_bucket[5m])) by (le))

histogram_quantile(0.75, sum(rate(nginx_request_time_bucket[5m])) by (le))

histogram_quantile(0.90, sum(rate(nginx_request_time_bucket[5m])) by (le))

histogram_quantile(1.00, sum(rate(nginx_request_time_bucket[5m])) by (le))

P.S. It is possible to compact these queries into a single one when using histogram_quantiles function from Prometheus-compatible query engine such as MetricsQL:

histogram_quantiles(
  "percentile",
  0.50, 0.75, 0.90, 1.00,
  sum(rate(nginx_request_time_bucket[5m])) by (le)
)

Note that the accuracy for percentiles calculated over Prometheus histograms highly depends on the chosen buckets. It may be hard to choose the best set of buckets for the given set of percentiles, so it may be better to use histograms with automatically generated buckets. See, for example, VictoriaMetrics histograms.