JVM Memory Problems

Question

Please help me out with a situation. I have CentOS with 4 GB RAM with Tomcat configured with around 1500MB memory. At some points my machine hangs out when it reaches max Threads configured on it i.e. 200. We have to restart the application server to make it work again. The params are:

JAVA_OPTS="-server -Xss128k -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -XX:NewRatio=3 -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -Dfile.encoding=UTF8"

The jvm mem details are pasted below. Heap

 PSYoungGen      total 435456K, used 73027K [0x00000000e0c00000, 0x00000000fb580000, 0x0000000100000000)
  eden space 435200K, 16% used [0x00000000e0c00000,0x00000000e5350c70,0x00000000fb500000)
  from space 256K, 0% used [0x00000000fb540000,0x00000000fb540000,0x00000000fb580000)
  to   space 256K, 88% used [0x00000000fb500000,0x00000000fb538600,0x00000000fb540000)
 PSOldGen        total 741376K, used 132680K [0x00000000a2400000, 0x00000000cf800000, 0x00000000e0c00000)
  object space 741376K, 17% used [0x00000000a2400000,0x00000000aa592030,0x00000000cf800000)
 PSPermGen       total 22912K, used 22880K [0x0000000092400000, 0x0000000093a60000, 0x00000000a2400000)
  object space 22912K, 99% used [0x0000000092400000,0x0000000093a58088,0x0000000093a60000)

Code Cache  [0x00002aaaab021000, 0x00002aaaab311000, 0x00002aaaae021000)
 total_blobs=1033 nmethods=670 adapters=317 free_code_cache=47372736 largest_free_block=12096

0x00002aaab41b7800 JavaThread ""http-apr-80"-exec-214" daemon [_thread_blocked, id=2636, stack(0x0000000043b06000,0x0000000043b27000)]

Answer 1

I don't think you have a memory problem here.

I think that you just ran out of available threads in tomcat's internal thread pool.
Tomcat cannot process requests anymore if its internal request processing thread pool does not have any free threads to do some more work and just waits till this situation changes (maybe forever).

My guess is that at least one servlet/jsp of one of your web applications never completes/returns.

Take a thread dump to confirm this and to easily identify the part of the software which causes this.