Hibernate, SessionFactoryObjectFactory and OutOfMemoryError:java heap space

Question

Where I work, we are experiencing problems with the JVM running out of heap-space in one of our applications. I have done some searching for the cause of this, including looking at a heap-dump with a profiler, but now I'm pretty much stuck.

First, a bit about the system in question: It is a Java application, using Spring and Hibernate, to keep records about organizations. The system is composed of a set of webservice clients that is used to retrieve data about organizations from the government institution responsible for this type of data. In addition, the system keeps a local database with such data, acting as a cache for the webservice calls, so that the first time information about an organization is requested, it is saved in a local relational database, and this is used for the retrieval of data for the following requests. Hibernate is used for communicating with this database.

The problem, as indicated earlier, is that after a period of time, the application start crashing with OutOfMemoryError: java heap space. I've looked at a heap-dump using Eclipse+MAT, and identified the culprit as being Hibernate's SessionFactoryObjectFactory, taking up approximately 85% of the allocated memory (all of it retained memory). I've found it a bit difficult to identify exactly what type of objects are kept within this. On the top level there is Glassfish WebappClassLoader, which contains the org.hibernate.impl.SessionFactoryObjectFactory. Contained in this is a org.hibernate.util.FastHashMap, which in turn contains a java.util.HashMap. This contains a number of entries, each of which containing a HashMap-entry, a org.hibernate.impl.SessionFactoryImpl and a String. The HashMap-entry in turn contains the same three objects, a HashMap-entry, a SessionFactoryImpl and a String, and this structure repeats itself a number of times. The SessionFactoryImpls contains a number of objects, most notably org.hibernate.persister.entity.SingleTableEntityPersister, which contains a number of Strings and HashMaps. Some of the Strings refer to variables in domain objects, and some contain sql-statements.

It seemed at first glance that this object was taking up uneccessary amounts of memory (the dump-file was 800MB, of which 650MB was occupied by SessionFactoryObjectFactory), so I enabled logging of object loading and unloading, and tried asking the system for data about an organization (through a webservice call from another system). What I noticed here was that there were a lot of messages for loading objects, but very few about unloaded objects (the only ones that were there were unloading of library objects). This lead me to believe that once an object (say an organization) was loaded into memory, it is never unloaded, which means that over time, the system will run out of memory. (Is this a fair assumption based on what was found in the log?)

Then, I tried finding the cause of this, but this was a lot harder. As objects loaded by Hibernate will live as long as their session lives, i tried altering the way sessions were handled, by replacing calls to Spring's HibernateDaoSupport#getSession(), to HibernateDaoSupport#getSessionFactory().getCurrentSession(). This had no appearant effect on the problem. I also tried adding calls to ...getCurrentSession().flush() and .clear() in the finally-block of some of the Dao-methods in question, also with no appearant effect. (The Dao-methods are all annotated with @Transactional, which should mean that a session should only be alive within the @Transactional-method, and consecutive calls to the method should get different sessions when calling getCurrentSession() (?))

So, now I'm pretty much stuck when it comes to coming up with other areas to check. Does anyone have an idea or some pointer about where to look and what to look for?

The heap-dump showed that there are a lot of instances of org.hibernate.impl.SessionFactoryImpl, is this as expected? (I would have thought that there should only be one instance of SessionFactory, or a few tops.)

Edit:

I think I've actually mananged to solve the problem:

It turned out that way dependencies to other objects were handled in the webservice-classes was the problem. This was solved by calling new ClassPathXmlApplicationContext(...) in the constructor of the webservice classes. This led to a lot of objects being loaded for each request (or at least each session), that were not unloaded again (mainly Hibernate's SessionFactoryImpl). I've changed the webservice-classes so they inject their dependency instead, and form what I've seen using profilers so far, the problem with multiple SessionFactoryImpl-objects has been solved.

I think the problem might have been worsened by upgrading from GlassFish 2.x to GlassFish 3.x, might be some differences as to how webservice-classes are instantiated.

Answer 1

I might as well add the solution to this problem in an answer, instead of just in the question itself:

The culprit here was how the loading of spring-beans was done in various objects, most notably in webservice-classes. This was done by calling

new ClassPathXmlApplicationContext(...)

In the individual webservice-classes. Doing this has the nasty side-effect of the objects loaded avoiding to be garbage collected (I guess because they are referenced by some of Spring's internals). It seems that a change in glassfish version did something to the instantiation of webservice-objects, leading to the calls to a lot more calls to new this, and thus a lot more junk objects occupying memory, until it fills up and crashes.

The solution to the problem was to move the calls to

new ClassPathXmlApplicationContext(...)

out into a different class, using the static factory-pattern, something like this:

public class ContextHolder {
    private static ClassPathXmlApplicationContext context;

    public static getSpringContext() {
        if (context == null) {
            context = new ClassPathXmlApplicationContext("applicationContext.xml");
        }
        return context;
    }
}

And calling this in the webservice-classes, instead of new-ing ClassPathXmlApplicationContext.

Update:

ClassPathXmlApplicationContext is Closeable/Autocloseable, so try-with-resourceis another possibility:

try (final ClassPathXmlApplicationContext applicationContext =
             new ClassPathXmlApplicationContext("applicationContext.xml")) {
    //do stuff
}