Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

A useful metric for determining when the JVM is about to get into memory/GC trouble

I have a scala data processing application that 95% of the time can handle the data thrown at it in memory. The remaining 5% if left unchecked doesn't usually hit OutOfMemoryError, but just gets into a cycle of major GCs that spikes the CPU, prevents background threads from executing and, if it does even finish, takes 10x-50x as long as when it has enough memory.

I've implemented system that can flush data to disk and treat the disk stream as if it was an in-memory iterator. It's usually an order of magnitude slower than memory, but sufficient for these 5% cases. I'm currently triggering by a heuristic of max size of a collection context that tracks the size of various collections involved in the data processing. This works, but really is just an adhoc empirical threshold.

I would much rather react to the JVM getting near the above bad state and flush to disk at that time. I've tried watching memory, but can't find the right combination of eden, old, etc. to reliably predict the death spiral. I've also tried just watching for frequency of major GCs but that also seems to suffer from having a wide range of "too conservative" to "too late".

Any resources for judging JVM health and detecting trouble states would be appreciated.

like image 992
Arne Claassen Avatar asked May 04 '15 22:05

Arne Claassen


People also ask

What are JVM metrics?

JVM Metrics. The Java Virtual Machine (JVM) Metrics tab allows you to view the heap memory, thread summary and garbage collection details of the JVM in which the Integration Server runs. Additionally, you can also view the operating system resource information such as CPU and Memory usage in time series.

How do I check JVM metrics?

The java. lang:type=Memory MBean exposes metrics for HeapMemoryUsage and NonHeapMemoryUsage so you can account for the JVM's combined heap and non-heap memory usage. You can also compare your physical server's system-level memory usage with JVM heap and non-heap usage by graphing these metrics on the same dashboard.

How do I monitor Java GC?

The typical CUI GC monitoring method involves using a separate CUI application called "jstat", or selecting a JVM option called "verbosegc" when running JVM. GUI GC monitoring is done by using a separate GUI application, and three most commonly used applications would be "jconsole", "jvisualvm" and "Visual GC".


1 Answers

One reliable way is to register a notification listener on GC events and check the memory health after all Full GC events. Directly after a full GC event, the memory used is your actual live set of data. If you at that point in time are low on free memory it is probably time start flusing to disk.

This way you can avoid false positives that often happens when you try to check memory with no knowledge of when a full GC has occurred, for example when using the MEMORY_THRESHOLD_EXCEEDED notification type.

You can register a notification listener and handle Full GC events using something like the following code:

// ... standard imports ommitted
import com.sun.management.GarbageCollectionNotificationInfo;

public static void installGCMonitoring() {
    List<GarbageCollectorMXBean> gcBeans = ManagementFactory.getGarbageCollectorMXBeans();
    for (GarbageCollectorMXBean gcBean : gcBeans) {
        NotificationEmitter emitter = (NotificationEmitter) gcBean;
        NotificationListener listener = notificationListener();
        emitter.addNotificationListener(listener, null, null);
    }
}

private static NotificationListener notificationListener() {
    return new NotificationListener() {
        @Override
        public void handleNotification(Notification notification, Object handback) {
            if (notification.getType()
                    .equals(GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION)) {
                GarbageCollectionNotificationInfo info = GarbageCollectionNotificationInfo
                        .from((CompositeData) notification.getUserData());
                String gctype = info.getGcAction();
                if (gctype.contains("major")) {
                    // We are only interested in full (major) GCs
                    Map<String, MemoryUsage> mem = info.getGcInfo().getMemoryUsageAfterGc();
                    for (Entry<String, MemoryUsage> entry : mem.entrySet()) {
                        String memoryPoolName = entry.getKey();
                        MemoryUsage memdetail = entry.getValue();
                        long memMax = memdetail.getMax();
                        long memUsed = memdetail.getUsed();
                        // Use the memMax/memUsed of the pool you are interested in (probably old gen)
                        // to determine memory health.
                    }
                }
            }
        }
    };
}

Cred to this article where we first got this idea from.

like image 75
K Erlandsson Avatar answered Sep 23 '22 02:09

K Erlandsson