How to tell if java application is close to running out of memory

Question

My simple goal: monitor the memory usage of a Java application so I can be warned when the application is getting dangerously close to throwing an OutOfMemoryError.

Yes, simple to state, but coming up with a correct solution seems very complicated. Some of the complicating factors are:

• There are different heap regions, each of which can throw an OutOfMemoryError:
  • The permgen space, which has it's own size limit (set via -XX:MaxPermSize=)
  • The overall heap space (set via -Xmx)
• The VM may allocate almost all of the heap before bothering to garbage collect. If the application uses a lot of soft references, then in fact this will surely happen. So just a high heap allocation percentage does not imply the application is near to throwing an OutOfMemoryError.
• It would be nice if System.gc() guaranteed that the VM would reclaim all possibly reclaimable object (unreferenced and/or weakly referenced object), but it doesn't. So invoking System.gc() and then Runtime.freeMemory() is not reliable.
• Objects that are queued for finalization take up memory, but (usually) are freed after they are finalized. So whether the finalizer thread has gotten to them or not affects the (apparent) memory usage (does the VM run the finalizer as a last desparate act before throwing OOM? Doesn't look like it.)
• Native code takes up memory as well and too much usage of it can lead to OOM (this is not a likely case in my specific application, but does add another complication to the overall picture).

So what is a good and reliable way to answer the question: Is my Java application getting to throwing an OutOfMemoryError?

Put another way, suppose application version X runs fine and has no memory leak, but version X + 1 has a slow unrecognized memory leak. I'd like to be alerted by this monitoring before version X + 1 throws an OutOfMemoryError, but I'd like the exact same monitoring to not give false positives for version X. There may be some tuning required in setting up this monitoring - that's OK.

One possible answer might be something like: what is the maximum, over the past N "full" GC runs, of the heap utilization immediately after the GC run? If this value exceeds X% of the total allocated memory, then sound the alarms.

The idea is to determine "application memory usage" in simple number like a percentage, or even something like LOW, MEDIUM, or HIGH, and then monitor this value.

The jstat command gives lots of relevant information, the problem is boiling it down to a simple answer and avoiding false positives (or negatives) caused by the complicating factors listed above.

Answer 1

If you watch a memory graph of a long-running application (collected with a tool like jconsole, for example) you'll see a characteristic sawtooth pattern: memory usage climbs, then is GC'd back to a baseline, and then it climbs again. For a healthy app, the peaks and valleys are in two straight horizontal lines. For a leaking app, though, the baseline climbs. That's really what you need to watch for: if each successive GC is less effective than the last, then something is rotten in Denmark.