Is there a way to record elapsed time using Prometheus

Question

I have a spring-boot application that I have instrumented using Prometheus to collect metric data. Prometheus has metrics such as Counter that is good for counting the number of times something has happened, Gauge which also keeps count, but can decrease, etc. Is there a metric that would allow me to track the request duration of something? For example, say I want to record the amount of time it takes between when an api call is made, and when the api returns with a response. How would I keep track of the time between when the call is made, and when the api response is received? Then, would it be possible to create a graph where on the Y coordinates, I can list the length of time (in seconds or milliseconds) that the response took; then on the X axis, have the time-stamp (the time when the metric was collected) be shown?

Answer 1

For timing event size, such as api latency, you probably want a Summary.

You can then calculate an aggregate average latency with:

  sum without (instance)(rate(my_summary_latency_seconds_sum[5m]))
/
  sum without (instance)(rate(my_summary_latency_seconds_count[5m])

Answer 2

As you mention spring-boot, you should really look at the Micrometer project. It provides a vendor neutral metric API with support for Prometheus. Micrometer has a specific meter primitive for timers:

Timer
   .builder("my.timer")
   .tags("uri", "/api/endpoint")
   .register(registry)
   .record(() -> {
     /* do something */
   });

// or
registry.timer("my.timer", "uri", "api/endpoint").record(() -> {
  /* do something */
});

For a Spring-Boot 1.5 project you'd use the micrometer-spring-legacy dependency. For Spring-Boot 2 (2.0.0.M7 as of this writing), Micrometer support is included OOTB with spring-boot-starter-actuator.

As you mention timing API requests, I assume you are using Spring MVC. There is automatic support for timing these. (You get a count for each URI, a sum of request duration and as an extra, a max for tracking the rolling max during a time window.)

In case you additionally wan't to calculate server-side request latency percentiles, you can enable the generation of histogram buckets, too.

Have a look at the documentation.