Java program (Tomcat) keeps eating memory (RES in top)

Question

I am running Tomcat on a 4-cpu and 32GB memory 64 bit machine (OS is CentOS 6.3). The Java option I start Tomcat with is -server -Xms1024m -Xmx1024m -XX:PermSize=512m -XX:MaxPermSize=512m

At the beginning, RES is just 810MB using top, and it keeps increasing. During this period, I run jmap -J-d64 -histo pID to check the Java memory heap, and I think the gc works fine, because heap peak is 510MB and around 200MB after gc. But when RES in top hits 1.1g, the CPU usage will exceeds 100% and Tomcat will hang.

Using jstack pid to see the dump when the CPU usage is 100%, a thread named "vm thread" eats up almost 100% CPU. I googled, it is the JVM gc thread. So my question is: Why does res keep growing when gc works fine? How could I resolve this problem? Thanks.

Answer 1

Might be a permgen leak. If you say your heap stays at ~500Mb, and -XX:MaxPermSize is set to 512Mb, full permgen will give you about 1gb of memory usage.

It could happen if you have lots (like, lots!) of jsps being loaded later in the lifecycle of your program, or you have used String.intern() a lot.

Follow this thread for further investigation: How to dump Permgen?

And this thread for tuning gc to sweep permgen What does JVM flag CMSClassUnloadingEnabled actually do?

Answer 2

If the garbage collection thread is spinning at 100% then it's likely that it is trying to perform garbage collection, but none of the objects can be collected so you are in a garbage collection death spiral where it keeps trying to run but not able to free any memory.

This may happen because you have a memory leak in your program, or because you are just not giving the vm enough memory to handle the number of objects you are loading during regular use. It sounds like you have plenty of headroom to increase the heap size. This may only prolong the amount of time before you hit the death spiral, or it may get you to a running state. You'll want to run a test for quite some time to ensure that you're not just delaying the inevitable.