Hibernate second level cache is closed while running a couple of spring tests

Question

I'm trying to write tests for application which is working based on Hibernate 5.3 and Spring Boot 2.1.3 and using Hibernate second level cache.

When I'm executing batch of test which are setting up spring context and trying to update some JPA entity, at some point getting the exception such this:

org.springframework.dao.InvalidDataAccessApiUsageException: Cache[default-update-timestamps-region] is closed; nested exception is java.lang.IllegalStateException: Cache[default-update-timestamps-region] is closed

at org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:370)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:255)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:536)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:746)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:714)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:533)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:304)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:135)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy244.save(Unknown Source)

I've the following configuration for Hibernate second level cache:

spring.jpa.properties.hibernate.cache.use_second_level_cache=true spring.jpa.properties.hibernate.cache.use_query_cache=true spring.jpa.properties.hibernate.cache.region.factory_class=org.hibernate.cache.jcache.JCacheRegionFactory spring.jpa.properties.javax.persistence.sharedCache.mode=ENABLE_SELECTIVE

and using Hibernate JCache as dependency.

From what I understand, org.hibernate.cache.jcache.JCacheRegionFactory reuse same instance of EhCache CacheManager for all contexts created by Spring Test but after some of time Spring shutdown cached context which is causing closing CacheManager and caches.

Previously, Hibernate (Hibernate EhCache module) provided org.hibernate.cache.ehcache.EhCacheRegionFactory factory which is creating new CacheManager every time and don't have problem described above.

Does anyone know how to create new CacheManager for each Spring test context and avoid using shared one ?

Answer 1

A possible workaround for this problem is adding @DirtiesContext like this to your class:

@DirtiesContext(classMode = DirtiesContext.ClassMode.BEFORE_CLASS)
public class SomeTestClass {
...
}

This will force Spring to create a new application context for all methods of this class. In my case, this resolved the issue.

Another approach is to ensure that Spring is aware of the Hibernate cache manager. This can be achieved like described in this blog post. However, this might not be possible in some cases.

Answer 2

The root GC cause lies in javax.cache.Caching which holds a static collection of CachingProvider-s shared between all Spring contexts if tests are run in the same JVM.

Spring contexts created during a test run share the same CachingProvider and therefore the same CacheManagers. When any of the contexts sharing a CachingProvider is closed, all related CacheManagers are closed too thus leaving remaining Spring contexts referring to the closed CachingProvider in an inconsistent state.

To address the issue, each request for a CacheManager should return a completely new instance not shared with other contexts.

I wrote a simple CachingProvider implementation that does just this and relies on existing CachingProviders. Please find the code below.

The base class:

import java.net.URI;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Properties;
import java.util.WeakHashMap;
import javax.cache.CacheManager;
import javax.cache.configuration.OptionalFeature;
import javax.cache.spi.CachingProvider;

/**
 * The abstract JCache compatible {@link CachingProvider} suitable for test purposes.
 *
 * <p>When using JCache and {@link org.hibernate.cache.jcache.JCacheRegionFactory}, {@link CachingProvider}-s
 * are shared between Spring contexts, which means that {@link CacheManager}-s are shared too. The class responsible
 * for storing loaded {@link CachingProvider}-s is {@link javax.cache.Caching}. If any cached Spring context is closed,
 * then all related {@link CacheManager}-s are closed as well, but since these {@link CacheManager}-s are shared with
 * remaining Spring contexts, we end up with in an inconsistent state.</p>
 *
 * <p>The solution is to make sure that each time a {@link CacheManager} for a particular config URI is requested, a new
 * instance not shared between Spring contexts is created</p>
 *
 * <p>The simplest approach is to create a new instance of {@link CachingProvider} for each {@link CacheManager} request
 * and manage them separately from {@link CachingProvider}-s loaded via {@link javax.cache.Caching}. This approach
 * allows reusing existing required {@link CachingProvider}-s and overcome any sharing issues.</p>
 *
 * <p>Tests relying on caching functionality MUST make sure that for regular caching the properties
 * {@code spring.cache.jcache.provider} and {@code spring.cache.jcache.config} are set and for 2nd-level cache
 * the properties {@code spring.jpa.properties.hibernate.javax.cache.provider} and
 * {@code spring.jpa.properties.hibernate.javax.cache.uri} are set. Please note that classpath URI-s for
 * the {@code spring.jpa.properties.hibernate.javax.cache.uri} property are supported by {@code hibernate-jcache} only
 * since 5.4.1, therefore with earlier versions this property should be set programmatically, for example via
 * {@link System#setProperty(String, String)}.</p>
 *
 * @see <a href="https://docs.jboss.org/hibernate/orm/5.2/userguide/html_single/Hibernate_User_Guide.html#caching-provider-jcache-cache-manager">Hibernate
 * JCache configuration</a>
 * @see org.hibernate.cache.jcache.JCacheRegionFactory
 * @see CachingProvider
 * @see javax.cache.Caching
 */
public abstract class AbstractTestJCacheCachingProvider implements CachingProvider {

    /**
     * The {@link CachingProvider}-s specific for a configuration {@link URI} for a specific {@link ClassLoader}.
     *
     * <p>All access MUST be handled in a <i>synchronized</i> manner.</p>
     */
    private final Map<ClassLoader, Map<URI, List<CachingProvider>>>
            classLoaderToUriToCachingProviders = new WeakHashMap<>();

    /**
     * {@inheritDoc}
     */
    @Override
    public CacheManager getCacheManager(URI uri, ClassLoader classLoader, Properties properties) {
        Objects.requireNonNull(uri, "The cache manager configuration URI must not be null.");
        Objects.requireNonNull(classLoader, "The class loader must not be null");

        final CachingProvider cachingProvider = createCachingProvider();
        synchronized (classLoaderToUriToCachingProviders) {
            classLoaderToUriToCachingProviders
                    .computeIfAbsent(classLoader, k -> new HashMap<>())
                    .computeIfAbsent(uri, k -> new ArrayList<>())
                    .add(cachingProvider);
        }
        return cachingProvider.getCacheManager(uri, classLoader, properties);
    }

    /**
     * Creates a {@link CachingProvider}.
     *
     * @return a created {@link CachingProvider}
     */
    protected abstract CachingProvider createCachingProvider();

    /**
     * {@inheritDoc}
     */
    @Override
    public ClassLoader getDefaultClassLoader() {
        return Thread.currentThread().getContextClassLoader();
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public URI getDefaultURI() {
        throw new UnsupportedOperationException("Please specify an explicit cache manager configuration URI.");
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public Properties getDefaultProperties() {
        return new Properties();
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public CacheManager getCacheManager(URI uri, ClassLoader classLoader) {
        return getCacheManager(uri, classLoader, null);
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public CacheManager getCacheManager() {
        throw new UnsupportedOperationException("The cache manager configuration URI must be specified.");
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public void close() {
        synchronized (classLoaderToUriToCachingProviders) {
            classLoaderToUriToCachingProviders.keySet().forEach(this::close);
        }
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public void close(ClassLoader classLoader) {
        Objects.requireNonNull(classLoader, "The class loader must not be null");

        synchronized (classLoaderToUriToCachingProviders) {
            // Process all CachingProvider collections regardless of the configuration URI.
            classLoaderToUriToCachingProviders
                    .getOrDefault(classLoader, Collections.emptyMap())
                    .values().stream().flatMap(Collection::stream)
                    // Close all CachingProvider resources since we are sure that CachingProvider-s are not shared
                    // or reused.
                    .forEach(CachingProvider::close);

            classLoaderToUriToCachingProviders.remove(classLoader);
        }
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public void close(URI uri, ClassLoader classLoader) {
        Objects.requireNonNull(uri, "The cache manager configuration URI must not be null");
        Objects.requireNonNull(classLoader, "The class loader must not be null");

        synchronized (classLoaderToUriToCachingProviders) {
            final Map<URI, List<CachingProvider>> uriToCachingProviders = classLoaderToUriToCachingProviders
                    .getOrDefault(classLoader, Collections.emptyMap());
            uriToCachingProviders
                    .getOrDefault(uri, Collections.emptyList())
                    // Close all CachingProvider resources since we are sure that CachingProvider-s are not shared
                    // or reused.
                    .forEach(CachingProvider::close);

            uriToCachingProviders.remove(uri);
        }
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public boolean isSupported(OptionalFeature optionalFeature) {
        // Find the first available CachingProvider and delegate the request to it.
        synchronized (classLoaderToUriToCachingProviders) {
            return classLoaderToUriToCachingProviders.values().stream().findFirst()
                    .flatMap(uriToCachingProviders -> uriToCachingProviders.values().stream().findFirst())
                    .flatMap(cachingProviders -> cachingProviders.stream().findFirst())
                    .map(cachingProvider -> cachingProvider.isSupported(optionalFeature))
                    .orElse(false);
        }
    }
}

The Ehcache based implementation:

import javax.cache.spi.CachingProvider;
import org.ehcache.jsr107.EhcacheCachingProvider;

/**
 * The test {@link CachingProvider} based on {@link EhcacheCachingProvider}.
 */
public class TestEhcacheJCacheCachingProvider extends AbstractTestJCacheCachingProvider {

    @Override
    protected CachingProvider createCachingProvider() {
        return new EhcacheCachingProvider();
    }
}