Why do the GC times increase steadily on a long running high volume Java app?

Question

I have a high volume Java application that handles a consistent load of 50000msgs/sec. It is tuned for high throughput using the following settings:

-Xmx3g -Xms3g -XX:NewSize=2g -Xss128k -XX:SurvivorRatio=6 -XX:TargetSurvivorRatio=90 -XX:+UseParallelGC -XX:ParallelGCThreads=12 -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError

I am finding that the young GC times steadily creeps up from 50ms when it starts to 200ms by the end of the day, although the frequency of GC runs remains the same.

If I try the same run using the ParNewGC collector, the GC times goes up at a much faster rate. Does anyone have any thoughts on this problem?

Answer 1

If you have a memory leak, or an in-memory cache that is gradually using more and more memory, these would have the effect of causing the GC to do more work tracing the reachable objects.

Other possibilities are:

• You have a non-heap memory leak, and that is resulting in increasing paging; i.e. copying of physical memory pages to disc and back.

• Some external process is consuming increasing amounts of memory, resulting in increasing paging.

• The nature of the objects generated in the young generation is changing over time so that more work is needed to trace the reachable objects. It may simply be that a larger proportion of them are surviving past the first collection.

Answer 2

It may be a combination of several factors:

• A real memory leak. ( Objects are still referenced by accident)
• More objects, or maybe a different distribution of the objects between the gc-pools.
• The heap is larger. Plot the gc-time and the heap size.
• Memory fragmentation. On OS level and/or the java heap, depending on the jvm.

Why don't you plot the number of objects, heap size and gc-time. (using a tool, or by quering MBeans) A plot should tell you the trend; increasing, decreasing or linear, and will help you to determine if there is a leak. If possible, also dump some statistics from your program, like the number of processed messages or active sessions etc.

Is it a real problem since your messages are not processed during gc?. Try the concurrent mark and sweep gc...

* -XX:+UseConcMarkSweepGC -XX:+UseParNewGC