JUnit tests fail when run together, but pass individually

Question

I have a bunch of JUnit tests that all function individually. Each one is a true standalone unit test - single class under test. No contexts are required. I can run them all individually or all together either in Eclipse or via maven / surefire-plugin.

I have since added a new Integration test which leverages the Spring Context, etc and uses the SpringJUnit4ClassRunner. As soon as I add this test to my suite, any test cases run after this class fail.

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = IntegrationTestConfiguration.class)
@DirtiesContext(classMode=ClassMode.AFTER_EACH_TEST_METHOD)
@ActiveProfiles("test")
public class ImportServiceIntegrationTest {
   ...
}

I'm not sure this has tremendous value, but I am posting my configuration class here as well:

@EnableAutoConfiguration(exclude = { WebMvcAutoConfiguration.class,
        DispatcherServletAutoConfiguration.class,
        EmbeddedServletContainerAutoConfiguration.class,
        WebSocketAutoConfiguration.class })
@ComponentScan(basePackages = "com.rtc.synchronize",
        excludeFilters = @ComponentScan.Filter(type = FilterType.REGEX, pattern="com\\.rtc\\.synchronize\\.config\\.AppConfig"))
@EnableJpaRepositories("com.util.veracode.rtc.synchronize")
@EntityScan("com.util.veracode.rtc.synchronize")
public class IntegrationTestConfiguration {
}

If my actual @Configuration classes will be of use, I can post those as well, although for brevity, I have avoided them (I'm not entirely sure how useful they would be).

I suspect that there is something maintained in the JVM (some static data) after the test class is terminated.

I am using Spring Cache annotations with the following config:

@Configuration
@EnableCaching(mode=AdviceMode.ASPECTJ)
public class CacheConfig extends CachingConfigurerSupport{

    /**
     * EhCache configuration.  Used to minimize calls to Veracode
     * 
     * @return
     */
    @Bean(destroyMethod="shutdown")
    public net.sf.ehcache.CacheManager ehCacheManager() {
        ...
        ...
    }
        ...
}

As soon as my integration test class finishes, my subsequent test throws the following error:

java.lang.IllegalStateException: The workItems Cache is not alive (STATUS_SHUTDOWN)
        at net.sf.ehcache.Cache$CacheStatus.checkAlive(Cache.java:4097)
        at net.sf.ehcache.Cache.checkStatus(Cache.java:2788)
        at net.sf.ehcache.Cache.get(Cache.java:1744)
        at org.springframework.cache.ehcache.EhCacheCache.get(EhCacheCache.java:65)
        at org.springframework.cache.interceptor.AbstractCacheInvoker.doGet(AbstractCacheInvoker.java:68)
        at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:461)
        at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:432)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:333)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:299)
        at org.springframework.cache.aspectj.AbstractCacheAspect.ajc$around$org_springframework_cache_aspectj_AbstractCacheAspect$1$2bc714b5(AbstractCacheAspect.aj:74)
        at com.synchronize.repository.rtc.WorkItemRepositoryImpl.findById(WorkItemRepositoryImpl.java:192)
        at com.synchronize.repository.rtc.WorkItemRepositoryImpl.findById(WorkItemRepositoryImpl.java:192)
        at com.synchronize.repository.rtc.WorkItemRepositoryImpl.getState(WorkItemRepositoryImpl.java:179)
        at com.synchronize.repository.rtc.WorkItemRepositoryImplTest.testGetState(WorkItemRepositoryImplTest.java:178)

So it is fairly clear to me that Spring is not cleaning something up after it is done (my subsequent class doesn't even load the Spring context - it is a plain vanilla Junit test!).

If I add <resueForks>false</reuseForks> to my surefire-plugin definition, all tests pass, but I am not happy about that solution/workaround. It slows down the build and it isn't respected in Eclipse - that is I simply run a JUnit test runner against an entire project in a single shot without it failing.

Do I have to do something special to ensure that Spring clears itself out of the JVM once the test case is complete? Why do I have some Spring configuraiton hanging around post my integration test?

Answer 1

I noticed in my test suites that some AspectJ based implementations have static fields that reference some spring bean, and this reference gets not updated when the profile or even the complete spring test context changes (I observed this for Spring-Security, but maybe there is a similar problem with @Cache)

My Workaround is to group the test cases by used spring configuration in different directories (for example src/test/javaSecurity) and also use a naming pattern to separate them. - The separated directories/source folders are used for eclipse, so can I click on the directory root folder and instruct eclipse to run all tests in this source folder. - The naming pattern is used for executing the test with maven (because surefire has the feature to select the executed tests by class naming pattern, but not by root folder)

• Directories/Patterns
  • src/test/java, pattern: *Test - contains tests with a spring configuration without spring-security
  • src/test/javaSecurity, pattern: *SecurityTest Spring tests that need an enabled spring security

maven

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>add-security-test-source</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-test-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>${basedir}/src/test/javaSecurity</source>
                </sources>
            </configuration>
        </execution>
    </executions>
</plugin>


<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <executions>
        <execution>
            <id>normal-test</id>
            <phase>test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>
                <excludes>
                    <exclude>**/Abstract*.java</exclude>
                    <exclude>**/*SecurityTest.java</exclude>
                </excludes>
                <includes>
                    <include>**/*Test.java</include>
                    <include>**/*Tests.java</include>
                </includes>
                <reportsDirectory>${project.build.directory}/surefire-reports/normaltest</reportsDirectory>
            </configuration>
        </execution>            
        <execution>
            <id>security-tests</id>
            <phase>test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>
                <excludes>
                    <exclude>**/Abstract*.java</exclude>
                </excludes>
                <includes>
                    <include>**/*SecurityTest.java</include>
                </includes>
                <reportsDirectory>${project.build.directory}/surefire-reports/security-test</reportsDirectory>
            </configuration>
        </execution>
    </executions>
    <configuration>
        <!-- just needed to prevent some bugs -->
        <excludes />
        <includes>
            <include>nothing</include>
        </includes>
        <reportsDirectory>${project.build.directory}/surefire-reports/nothing</reportsDirectory>
    </configuration>
</plugin>