Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Micrometer/Prometheus How do I keep a gauge value from becoming NaN?

I am trying to monitor logged in users, i am getting the logged in user info by calling api, this is the code i have used,

public class MonitorService {
    private InfoCollectionService infoService;
    public MonitorService(InfoCollectionService infoService) {
        this.infoService = infoService
    }

    @Scheduled(fixedDelay = 5000)
    public void currentLoggedInUserMonitor() {
        infoService.getLoggedInUser("channel").forEach(channel -> {
            Metrics.gauge("LoggedInUsers.Inchannel_" + channel.getchannelName(), channel.getgetLoggedInUser());
        });
    }
}

And i see the values in Prometheus, the problem is after a few seconds, the value become NaN, i have read that Micrometer gauges wrap their obj input with a WeakReference(hence Garbage Collected ).I don't know how to fix it.If anybody knows how to fix this it would be great.

like image 615
Ipkiss Avatar asked Jun 15 '18 16:06

Ipkiss


People also ask

Why is my gauge reporting NaN or disappearing?

If you see your gauge reporting for a few minutes and then disappearing or reporting NaN, it almost certainly suggests that the underlying object being gauged has been garbage collected.

What are gauge values?

A gauge is a metric that represents a single numerical value that can arbitrarily go up and down. Gauges are typically used for measured values like temperatures or current memory usage, but also "counts" that can go up and down, like the number of concurrent requests.

What is gauge in Prometheus?

A gauge in Prometheus is represented by a 64-bit floating point number. That means it can store very large or small decimal numbers, either positive or negative.


1 Answers

This is a shortcoming in Micrometer that I would like to fix eventually.

You need to keep the value in a map in the meantime so it avoid the garbage collection. Notice how we then point the gauge at the map and us a lambda to pull out the value to avoid the garbage collection.

public class MonitorService {
    private Map<String, Integer> gaugeCache = new HashMap<>();
    private InfoCollectionService infoService;
    public MonitorService(InfoCollectionService infoService) {
        this.infoService = infoService
    }

    @Scheduled(fixedDelay = 5000)
    public void currentLoggedInUserMonitor() {
        infoService.getLoggedInUser("channel").forEach(channel -> {
            gaugeCache.put(channel.getchannelName(), channel.getgetLoggedInUser());
            Metrics.gauge("LoggedInUsers.Inchannel_" + channel.getchannelName(), gaugeCache, g -> g.get(channel.getchannelName()));
        });
    }
}

I would also recommend using tags for the various channels:

Metrics.gauge("loggedInUsers.inChannel", Tag.of("channel",channel.getchannelName()), gaugeCache, g -> g.get(channel.getchannelName()));
like image 73
checketts Avatar answered Sep 18 '22 07:09

checketts