Classloader is not garbage collected, even without GC roots

Question

We have a complex application running under Glassfish V2.1.1. In order to be able to load our code dynamically, we have implemented a CustomClassloader which is able to redefine classes. The behaviour is quite easy: when a dynamically loaded class has change, the current instance of the CustomClassloader is "dropped" and a new one is create to redefine the needed classes.

This works well except that after a few number of time the same class has been reloaded (hence each time a new CustomClassloader is created), we get a PermGen space error because the other instances of CustomClassloader are not garbage collected. (There should be only one single instance of this class)

I tried different methods to track down where the leak is:

1. visualvm => I make a heap dump and extract all the instances of the CustomClassloader. I can see that none of them have been finalized. When I check for the nearest GC root, visualvm tells me that there is none (except for the last instance, because it is the 'real' used one).
2. jmap/jhat => It gives me almost the same result: I see all the instances of CustomClassloader and then when I click on the link to see where are the references on one of them, I get a blank page meaning there is none...
3. Eclipse Memory Analyzer Tool => I get a strange result when I run the following OQL query: SELECT c FROM INSTANCEOF my.package.CustomClassloader c There is only one result, indicating there is only one single instance which is obviously not correct.

I also checked this link and implemented some resources release when a new CustomClassloader is created, but nothing changes: the PermGen memory is still increasing.

So I'm probably missing something and the difference between points (1-2) and (3) shows something I do not understand. Where can I look to get an idea of what's wrong ? Since all the tutorials I followed show how to search the leaking references by using the "Search nearest GC root" feature (and in my case there is none), I don't know how I can track the error.

EDIT 1: I uploaded an example of the heap dump here. The ClassLoader not being unloaded can be selected in visualvm with the following query: select s from saierp.core.framework.system.SAITaskClassLoader s One can see that there are 4 instances and the three first should have been collected because there is no GC root... There must be a reference somewhere but I don't know how I can search for it. Any hint is welcomed :)

EDIT 2: After some deeper tests I see a very strange pattern. The leak seems to depend on the data that are being loaded by OpenJPA: if no new data is loaded, then the classloader can be GCed, otherwise it is not. Here is the code I use when a create a new SAITaskClassLoader to 'clear' the old one:

PCRegistry.deRegister(cl);
LogFactory.release(cl);
ResourceBundle.clearCache(cl);
Introspector.flushCaches();

= Pattern 1 (Classloader is GCed): =

1. New SAITaskClassLoader
2. Load Data D1, D2, ..., Dn
3. New SAITaskClassLoader
4. Load Data D1, D2, ..., Dn
5. ...

= Pattern 2 (Classloader is NOT GCed): =

1. New SAITaskClassLoader
2. Load Data D1, D2, D3
3. New SAITaskClassLoader
4. Load Data D3, D4, D5
5. New SAITaskClassLoader
6. Load Data D5, D6, D7
7. ...

In all cases, the SAITaskClassLoader that have been cleared have no GC root. We are using OpenJPA 1.2.1.

Thanks & Best regards

Answer 1

Without snippets of the source code of CustomClassLoader or the actual heap dumps, it will be very hard to track down the problem. Your CustomClassLoader can't be a singleton. If it is, your design can't work (or i missed something).

You need to get the list of ClassLoader instances of type CustomClassLoader and track down the references to these objects.

These posts may help you on how to analyze it further and get into the dark detaisls of hunting down ClassLoader leaks:

• Debugging Dynamic ClassLoaders in Heap Dumps
• How to analyze leaky web-apps
• A couple of typical causes of ClassLoader leaks

Answer 2

Garbage collection of classloaders is an exceptionally tricky business. Using JProfiler, I see the following chain of incoming references to the currently active custom class loader:

This shows that you have a static field "singleInstance" in your custom classloader that references the class loader itself. You should try to clear that field on redeployment in order to make it easier for the VM to collect the classloader.

A note about the result you got with Eclipse MAT: It removes all objects that are not strongly reachable. JProfiler also does this by default. So the three previous classloaders should be garbage collected but they are not, due to the special rules that the JVM has for classloader GC which are not captured by the standard references in the heap.

Disclaimer: My company develops JProfiler

Answer 3

Finally, I can close this bug, since it seems to be linked with OpenJPA and non-parametrized queries. Another thread to look at: Custom ClassLoader not garbage collected