Memory Fully utilized by Java ConcurrentHashMap (under Tomcat)

Question

This is a memory stack (serves as a cache) that consist of nothing but a static ConcurrentHashMap (CHM).

All incoming HTTP request data are store in this ConcurrentHashMap. And there is a asynch scheduler process that takes the data from the same ConcurrentHashMap and remove the key.value after storing them into the Database.

This system runs fine and smooth but just discover under following criteria, the memory was fully utilized (2.5GB) and all CPU time was taken to perform GC:

-concurrent http hit of 1000/s

-maintain the same concurrent hit for a period of 15 minutes

The asynch process log the remaining size of the CHM everytime it writes to database. The CHM.size() maintain at around Min:300 to Max:3500

I thought there is a Memory Leak on this application. so i used Eclipse MAT to look at the Heap Dump. After running the Suspect Report, i got these comments from MAT:

One instance of "org.apache.catalina.session.StandardManager" loaded by "org.apache.catalina.loader.StandardClassLoader @ 0x853f0280" occupies 2,135,429,456 (94.76%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "".

3,646,166 instances of java.util.concurrent.ConcurrentHashMap$Segment retain >= 2,135,429,456 bytes. 

and

Length    # Objects      Shallow Heap      Retained Heap  0         3,646,166      482,015,968       >= 2,135,429,456  

The length 0 above i translate it as empty length record inside the CHM (each time i call CHM.remove() method). It is consistent to the number of record inside the database, 3,646,166 records was inside the database when this dump was created

The strange scenario is: if i pause the stress test, the utilization in Heap Memory will gradually release down to 25MB.This takes about 30-45 minutes. i have re-simulate this application and the curves looks similar to the VisualVM Graph below:

Heres the questions:

1) Does this looks like a Memory Leak?

2) Each remove call remove(Object key, Object value) to remove a <key:value> from CHM, does that removed object get GC?

3) Is this something to do with the GC settings? i have added the following GC parameters but without help:

-XX:+UseParallelGC  -XX:+UseParallelOldGC  -XX:GCTimeRatio=19  -XX:+PrintGCTimeStamps  -XX:ParallelGCThreads=6  -verbose:gc 

4) Any idea to resolve this is very much appreciated! :)

NEW 5) Could it be possible because all my reference are hard reference? My understanding is as long as the HTTP session is ended, all those variables that is not static are now available for GC.

NEW Note I tried replace the CHM with ehcache 2.2.0, but i get the same OutOfMemoryException problem. i suppose ehcache is also using ConcurrentHashMap.

Server Spec:

-Xeon Quad core, 8 threads.

-4GB Memory

-Windows 2008 R2

-Tomcat 6.0.29

Answer 1

This problem has bug me for a bad 7 days! And finally i found out the real problem! Below are the tasks on what i have tried but failed to solve the OutOfMemory Exception:

-change from using concurrenthashmap to ehcache. (turns out ehcache is also using ConcurrentHashMap)

-change all the hard reference to Soft Reference

-Override the AbstractMap along side with concurrnetHashMap as per suggest by Dr. Heinz M. Kabutz

The million dollar question is really "why 30-45 minutes later, memory starting to release back to the heap pool?"

The actual root cause was because there is something else still holding the actual variable session, and the culprit is the http session within tomcat is still active! Hence, even though the http session was completed, but if the timeout setting is 30 minutes, tomcat will hold the session information for 30 minutes before JVM can GC those. Problem solve immediately after changing the timeout setting to 1 minute as testing.

$tomcat_folder\conf\web.xml  <session-config>     <session-timeout>1</session-timeout> </session-config> 

Hope this will help anyone out there with similar problem.

Answer 2

I think you're using too much session data that won't fit at once in memory. Try this one:

1. Edit bin/setenv.sh or wherever the JVM args are set on your Tomcat launcher :

   Append -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true

   e.g.

   # Default Java options if [ -z "$JAVA_OPTS" ]; then         JAVA_OPTS="-server -Djava.awt.headless=true -XX:MaxPermSize=384m -Xmx1024m -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true" fi 

2. Edit conf/context.xml, before </Context> add this:

   <Manager className="org.apache.catalina.session.PersistentManager"         maxIdleBackup="60" maxIdleSwap="300">     <Store className="org.apache.catalina.session.FileStore"/> </Manager> 

Restart Tomcat and your problem should be gone, since it'll store your sessions using the filesystem instead.

In my view setting session-timeout = 1 is a workaround that masks the root of problem, and is unusable in most apps where you actually need a big enough session-timeout. Our (Bippo's) apps usually have a session-timeout of 2880 minutes i.e. 2 days.

Reference: Tomcat 7.0 Session Manager Configuration