Unreachable objects are not garbage collected from heap

Question

I'am struggling with unreachable objects in my JVM heap (Java 1.7). As you can see from the picture (all classes on the picture are unreachable), we have more than 74 % objects with no reference, so It should be garbagged collected. This state becomes after 3 weeks uptime on our tomcat 7 server where run only Probe monitoring app, tomcat manager and our webapp which is probably source of the problem.

Our application is based on JSF 1.2 with state saving on client which is what you see in picture below - char arrays with ViewSaveState mostly. When I manually run GC from jVisualVM It removes all unreachable objects and everything is ok until 3 weeks when the heap reaches its limit.

How is it possible that some objects aren't cleaned up?

Our JVM params

-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=29001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=
-Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true
-Xms320m
-Xmx2500m
-XX:MaxPermSize=500m
-XX:PermSize=96m
-verbose:gc
-Xloggc:/var/log/gc.log
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-Xdebug -Xrunjdwp:transport=dt_socket,address=1044,server=y,suspend=n
-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC

STACKTRACES for OutOfMemoryError

I think that cause is hidden somewhere else, stacktraces came from different part of app. There could be some leak, but stacktraces report only last component which claims some memory in time when there isn't any.

    java.lang.OutOfMemoryError: Java heap space
            at java.util.LinkedHashMap.createEntry(LinkedHashMap.java:442)
            at java.util.HashMap.addEntry(HashMap.java:888)
            at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:427)
            at java.util.HashMap.put(HashMap.java:509)
            at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResourceBundle.java:134)
            at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(OpenListResourceBundle.java:113)
            at sun.util.resources.OpenListResourceBundle.handleGetObject(OpenListResourceBundle.java:74)
            at sun.util.resources.TimeZoneNamesBundle.handleGetObject(TimeZoneNamesBundle.java:75)
            at java.util.ResourceBundle.getObject(ResourceBundle.java:389)
            at java.util.ResourceBundle.getObject(ResourceBundle.java:392)
------------------
Exception in thread "Timer-22" Exception in thread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" java.lang.OutOfMemoryError: Java heap space
Exception in thread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" java.lang.OutOfMemoryError: Java heap space
------------------
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2219)
        at java.util.ArrayList.grow(ArrayList.java:242)
        at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216)
        at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208)
        at java.util.ArrayList.add(ArrayList.java:440)
        at org.hibernate.loader.Loader.instanceNotYetLoaded(Loader.java:1468)
        at org.hibernate.loader.Loader.getRow(Loader.java:1355)
        at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:611)
        at org.hibernate.loader.Loader.doQuery(Loader.java:829)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
         at org.ajax4jsf.component.AjaxActionComponent.broadcast(AjaxActionComponent.java:55)
        at org.ajax4jsf.component.AjaxViewRoot.processEvents(AjaxViewRoot.java:329)
        at org.ajax4jsf.component.AjaxViewRoot.broadcastEventsForPhase(AjaxViewRoot.java:304)
        at org.ajax4jsf.component.AjaxViewRoot.processPhase(AjaxViewRoot.java:261)
        at org.ajax4jsf.component.AjaxViewRoot.processApplication(AjaxViewRoot.java:474)
        at org.apache.myfaces.lifecycle.InvokeApplicationExecutor.execute(InvokeApplicationExecutor.java:32)
        at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:103)
        at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:76)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:183)
        ... 74 more
--------------
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.nio.ByteBuffer.wrap(ByteBuffer.java:350)
        at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:137)
        at java.lang.StringCoding.decode(StringCoding.java:173)
        at java.lang.String.<init>(String.java:443)
        at com.ibm.db2.jcc.a.a.a(a.java:632)
        at com.ibm.db2.jcc.a.a.a(a.java:355)
        at com.ibm.db2.jcc.am.fc.e(fc.java:682)
        at com.ibm.db2.jcc.am.fc.k(fc.java:1481)
        at com.ibm.db2.jcc.am.ResultSet.getTimestampX(ResultSet.java:1075)
        at com.ibm.db2.jcc.am.ResultSet.getTimestamp(ResultSet.java:1034)

Answer 1

One possibility is that you are overloading the JVM with pathological behavior in finalize() methods. If you have classes that override Object.finalize() the JVM has to do a surprising amount of work in order to actually clean them up (and, in turn, clean up all their referenced objects). If you create such objects faster than the garbage collector can process them you'll quickly run into trouble.

This article walks through a pathological finalizer example in detail, but to summarize:

1. An object with a finalize() method falls out of scope and (conceptually) becomes eligible for GC.
2. Unlike a normal object which can simply be freed, the JVM holds a special reference to the object via a Finalizer, preventing these objects from being trivially collected. Even short-lived objects survive the initial GC and move into longer-lived sections of the heap if associated with a finalize().
3. Now these objects will be processed by the dedicated Finalizer thread, which will call .finalize() on each object in turn. Objects that haven't yet had their turn remain on the heap though they're unreachable.
4. Once the object is processed by the Finalizer thread this last reference is removed and the object can finally actually be GC'ed. Since the object survived one or more collection rounds it may take some time for the GC to get around to it.
5. Any objects the finalizable object referenced in turn are only now eligible for collection.

If your .finalize() methods take a particularly long time or if you're creating a large number of such objects the Finalizer thread can't keep up with the demand and objects will continue to queue up, eventually filling your entire heap.

There are other possible explanations, but over-use of finalize() is a likely reason. Effective Java Item 7 strongly discourages finalizers:

Finalizers are unpredictable, often dangerous, and generally unnecessary. Their use can cause erratic behavior, poor performance, and portability problems....

Providing a finalizer for a class can, under rare conditions, arbitrarily delay reclamation of its instances.

Answer 2

This question might be related to this one

Java heap overwhelmed by unreachable objects

I found it as I'm facing the same problem in an IBM JVM 5 runninf on AIX 6.1

The only amount of retained heap that grows consistently between two fullgcs is the unreachable objects, marked, if I'm not wrong, with ROOT as the dominator in Eclipse MAT.

To get these dumps, I configureg the JVM to create a sysdump after a fullgc using the Xdump option (only for IBM JVMs, I think)

Hope this helps, and if anybody knows if the GC verbose log leaves a trace of the unreachable objects that couldn't be cleaned, please let me know!

Carlos