How to remove no longer valid Gauges

Question

I use the Prometheus Java Client to export session information of my application. We want to show how long sessions have been idle.

The problem is that we have a maximum of 1000 sessions and sessions are removed after a certain period. Unfortunately they do not disappear from Prometheus:

My code looks like this:

static final Gauge sessionInactivity = Gauge.build()
    .name("sessions_inactivity_duration")
    .labelNames("internal_key", "external_key", "browser")
    .help("Number of milliseconds a certain session has been inactive")
    .register();

sessionInactivity.labels(internalKey, externalKey, browser).set(inactivityTime);

I tried to do sessionInactivity.clear() during scrapes but obviously this does not empty the content of the Gauge.

Answer 1

The Gauge class has a remove method which has the same signature as the labels method. For your specific example, removing the metrics associated with that gauge would look like this

sessionInactivity.remove(internalKey, externalKey, browser);

The client library documentation states:

Metrics with labels SHOULD support a remove() method with the same signature as labels() that will remove a Child from the metric no longer exporting it, and a clear() method that removes all Children from the metric. These invalidate caching of Children.

Answer 2

This sort of per-request handling is not suited to a metric system like Prometheus. This would be considered profiling, for which something more custom would be in order.

It's also recommend to export the timestamp for this sort of thing, not how long ago it was. This is resilient to the thing updating the time no longer updating, and you can do the subtraction from time() on the Prometheus end.

Answer 3

Since you are registrating the gauge, you need to remove the gauges which are expired from the registry.

protected MeterRegistry registry;

  ...
  static final Gauge sessionInactivity = Gauge.build()
    .name("sessions_inactivity_duration")
    .labelNames("internal_key", "external_key", "browser")
    .help("Number of milliseconds a certain session has been inactive")
    .register(registry); // use registry

  ...
  // remove expired gauges from registry
  registry.remove(sessionInactivity);