Memory Leak on stop or redeploy - Spring 3.1.2, Hibernate 4.1.0, Spring Data-Jpa 1.1.0, Tomcat 7.0.30

Question

EDIT 1

2013/06/07 - While I'm still having this problem, once again, it is still only affecting redeploys. Since the original question was posted, we've upgraded a few things. Here are our new version (which still exhibit the issue at hand):

<properties>
    <!-- Persistence and Validation-->
    <hibernate.version>4.1.0.Final</hibernate.version>
    <hibernate.jpa.version>1.0.1.Final</hibernate.jpa.version>
    <javax.validation.version>1.0.0.GA</javax.validation.version>
    <querydsl.version>2.2.5</querydsl.version>
    <spring.jpa.version>1.2.0.RELEASE</spring.jpa.version>
    <spring.ldap.version>1.3.1.RELEASE</spring.ldap.version>

    <!-- Spring and Logging -->
    <spring.version>3.1.3.RELEASE</spring.version>
    <spring.security.version>3.1.3.RELEASE</spring.security.version>
    <slf4j.version>1.6.4</slf4j.version>
    <jackson.version>1.9.9</jackson.version>

    <cglib.version>3.0</cglib.version>
</properties>

As you can see, it's basically just a Spring Framework bump and Spring (Data) Jpa bump. We also moved up to Tomcat 7.0.39. CGLIB (which wasn't mentioned before but was included) has also been bumped to 3.0

Here are some of the things I've tried to remedy the problem at hand with no luck:

1. Changed up our POM (using Maven) to set our database driver to have its scope as provided
2. Added dependency for SLF4J to included jul-to-slf4j as it was mentioned that there could be a potential leak here
3. Tried out a Classload Leak Protector (https://github.com/mjiderhamn/classloader-leak-prevention). This appeared to have worked (as it spammed my logs during undeploy/shutdown by listing a ton of things it was cleaning up) but after using VisualVM, my perm gen space was not reclaimed. This may be useful for others...
4. Tried to list all the exclusions in my POM by using Maven's dependency analysis tool (In IntelliJ in the Maven Projects window, you can right click the dependencies and select 'Show Dependencies' and Shift-Delete all the red dependencies). This didn't help, but it decrease my WAR file size by about a MB or so.

5. Refactored the JPA Persistence configuration as follows (notice the comments) based upon an unresolved bug report from Spring (https://jira.springsource.org/browse/SPR-9274):

   @Bean
   public PlatformTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) { // Important line (notice entityManagerFactory is 'provided/autowired'
       return new JpaTransactionManager(entityManagerFactory);
   }
   
   @Bean
   public EntityManagerFactory getEntityManagerFactory(DataSource dataSource) { // Important line (notice dataSource is 'provided/autowired'
   
       LocalContainerEntityManagerFactoryBean factoryBean = new LocalContainerEntityManagerFactoryBean();
       factoryBean.setDataSource(dataSource);
       factoryBean.setPackagesToScan("my.scanned.domain");
   
       AbstractJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
       vendorAdapter.setGenerateDdl(true);
       vendorAdapter.setShowSql(false);
       vendorAdapter.setDatabase(Database.POSTGRESQL);
       vendorAdapter.setDatabasePlatform("org.hibernate.dialect.PostgreSQL82Dialect");
   
       factoryBean.setJpaVendorAdapter(vendorAdapter);
   
       Map<String, Object> properties = new HashMap<>();
       properties.put("hibernate.ejb.naming_strategy", "org.hibernate.cfg.ImprovedNamingStrategy");
       properties.put( "hibernate.bytecode.provider", "cglib" );   // Suppose to help java pergem space issues with hibernate
   
       factoryBean.setPersistenceProvider(new HibernatePersistence());
       factoryBean.setJpaPropertyMap(properties);
       factoryBean.setPersistenceUnitName("myPersistenace");
       factoryBean.afterPropertiesSet();
   
       return factoryBean.getObject(); // Important line
   }
   
   @Bean
   public PersistenceExceptionTranslator getHibernateExceptionTranslator() { // Required
       return new HibernateExceptionTranslator();
   }
   
   @Bean
   public DataSource getDataSource() {
       JndiDataSourceLookup lookup = new JndiDataSourceLookup();
       DataSource dataSource = lookup.getDataSource("java:comp/env/jdbc/myLookup");
   
       lookup = null;
   
       return dataSource;
   }
   

6. Created a 'ThreadImmolator' as per https://stackoverflow.com/a/15710827/941187 in the following SO Question: 'Is this very likely to create a memory leak in Tomcat?'. This appeared to do nearly the same thing as the TreadLocal Leak Detection above -- I didn't have the Tomcat Leak detector complaining; however, perm gen space was still not recovered after redeploy.
   • My first attempt was to add a @PreDestory in my WebConfig (same bean that has @EnableWebMvc) to attempt to trigger it on close. Perm. Gen remained.
   • My second attempt was to subclass the ContextLoaderListener, override the ContextDestoryed() and inline the functions. Perm. Gen remained.
7. Since the ThreadImmolator didn't help (or appeared not to help), I tried the solution suggested in https://stackoverflow.com/a/16644476 on the following SO Question: 'How to clean up threadlocals'. This lead me to try: 'https://weblogs.java.net/blog/jjviana/archive/2010/06/09/dealing-glassfish-301-memory-leak-or-threadlocal-thread-pool-bad-ide' and 'http://blog.igorminar.com/2009/03/identifying-threadlocal-memory-leaks-in.html'.

At this point, I have run out of ideas.

I have also tried learning about Heap analysis using this as a starting point (http://cdivilly.wordpress.com/2012/04/23/permgen-memory-leak/). I can find the class loader that was not cleaned up and I can see that it is still referencing all of the classes related to Spring. I've also tried searching for org.springframework.core.NamedThreadLocal and I can still see them showing up in the Heap after taking a dump after running the ThreadImmolator, Thread Leak Preventor and the other 'heavy-handed-solutions' that I tried above.

Perhaps the above information may help someone, but I will continue to revisit this SO with new information or if I solve the issue.

Problem

The application has no issues running for days on end on the production server, but when I perform a deploy for updates, the Tomcat Manager program will complain about leaks (if I click find leaks). If I perform 6-10 deploys, eventually Tomcat runs out of memory with a PermGen memory error and I need to restart the Tomcat service and everything returns to normal.

When I run/debug the application locally and perform some actions that require access through Jpa/Hibernate (ie, I login or request a List from a JpaRepository) and then shutdown the application, I receive the following messages in my debug output from Tomcat:

Oct 03, 2012 2:55:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{public abstract java.lang.Object org.springframework.data.repository.CrudRepository.findOne(java.io.Serializable)=java.lang.Object@842e211}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Oct 03, 2012 2:55:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{public abstract java.util.List org.springframework.data.jpa.repository.JpaRepository.findAll()=java.lang.Object@842e211}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Oct 03, 2012 2:55:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{public abstract java.lang.Iterable org.springframework.data.querydsl.QueryDslPredicateExecutor.findAll(com.mysema.query.types.Predicate)=java.lang.Object@842e211}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Oct 03, 2012 2:55:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{public abstract data.domain.UserAccount UserAccountRepository.findByUserName(java.lang.String)=java.lang.Object@842e211}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

etc, etc.

Configuration

Spring is configured via annotations and I'm also using Postgres 8.4 as the database backend.

JPA is configured via annotations (jpa-repository-context.xml just says to look for this class):

@Configuration
@EnableTransactionManagement
@ImportResource( "classpath*:*jpa-repository-context.xml" )
@ComponentScan( basePackages = { "data.repository" } )
public class PersistenceJpaConfig
{
    @Bean
        public LocalContainerEntityManagerFactoryBean entityManagerFactory()
        {
            LocalContainerEntityManagerFactoryBean factoryBean = new LocalContainerEntityManagerFactoryBean();
            factoryBean.setDataSource( dataSource() );
            factoryBean.setPackagesToScan( new String[] { "data.domain" } );

            // Setup vendor specific information. This will depend on the chosen DatabaseType
            HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
            vendorAdapter.setGenerateDdl( true );
            vendorAdapter.setShowSql( false );
            vendorAdapter.setDatabasePlatform( "org.hibernate.dialect.PostgreSQL82Dialect" );

            factoryBean.setJpaVendorAdapter( vendorAdapter );

            Map<String, Object> properties = new HashMap<String, Object>();
            properties.put( "hibernate.ejb.naming_strategy", "org.hibernate.cfg.ImprovedNamingStrategy" );

            factoryBean.setJpaPropertyMap( properties );

            return  factoryBean;
        }

        @Bean
        public DataSource dataSource()
        {
            JndiDataSourceLookup lookup = new JndiDataSourceLookup();
            DataSource dataSource;

            dataSource = lookup.getDataSource( "java:comp/env/jdbc/postgres" );


            return dataSource;
        }

        @Bean
        public PlatformTransactionManager transactionManager()
        {
            JpaTransactionManager transactionManager = new JpaTransactionManager();
            transactionManager.setEntityManagerFactory( entityManagerFactory().getObject() );

            return transactionManager;
        }
}

Example Repository:

public interface UserAccountRepository extends JpaRepository<UserAccount, Long>, QueryDslPredicateExecutor<UserAccount> {
}

All of my Repositories are accessed through a Service class which is registered as an @Component in Spring. This is done to remove repository access from Spring controllers:

@Component
public class UserAccountService {

    @Autowired
    private UserAccountRepository userAccountRepository;

    public List<UserAccount> getUserAccounts() {
        return userAccountRepository.findAll();
    }
    ...
}

And here are the versions for the various components in use in Maven's pom.xml:

<properties>
        <!-- Persistence and Validation-->
        <hibernate.version>4.1.0.Final</hibernate.version>
        <hibernate.jpa.version>1.0.1.Final</hibernate.jpa.version>
        <javax.validation.version>1.0.0.GA</javax.validation.version>
        <querydsl.version>2.2.5</querydsl.version>
        <spring.jpa.version>1.1.0.RELEASE</spring.jpa.version>

        <!-- Spring and Logging -->
        <spring.version>3.1.2.RELEASE</spring.version>
        <spring.security.version>3.1.2.RELEASE</spring.security.version>
        <slf4j.version>1.6.4</slf4j.version>
        <jackson.version>1.9.9</jackson.version>

        <!-- Testing Suites -->
        <selenium.version>2.24.1</selenium.version>
</properties>

Questions

1. What is causing the memory leak and how do I fix it?
2. How would I go about debugging this particular issue?
3. Is there anything in my configuration set that could be improved?

I have really run out of ideas on solving this problem.

Answer 1

I think you may have two kinds of leaks going on at the same time. Spring is warning you about a normal "heap" memory leak. This has not caused you a problem yet, because ... your redeployment is also causing excessive PermGen usage, and this problem is hitting you first. For info on the second kind of leak see Dealing with "java.lang.OutOfMemoryError: PermGen space" error [Thanks duffymo]

[update]

Since you say the suggestions in the above link did not help, the only other suggestions I can think of are:

• make sure your spring beans are cleaning themselves up when spring shuts down
• every bean that allocates a resource (anything unlikely to be garbage collected) in the constructor or init method should have a destroy method to deallocate it
• this is especially true for any beans that reference any classes in the spring-data module, catalina is complaining about a class in this module
• increase your permgen space. Even if this suggestion did not fix the problem adding something like the following should make these failures occur less frequently.

Try -XX:MaxPermSize=256m and if it persists, try -XX:MaxPermSize=512m

The ultimate uber-brute-force approach is to gradually strip things out of your app until the problem goes away. That will help you narrow it down to the point where you can identify whether it is an issue with your code or a bug in Spring, Hibernate etc