How can I use a custom runner when using categories in Junit?

Question

I have a bunch of JUnit tests that extend my base test class called BaseTest which in turn extends Assert. Some of my tests have a @Category(SlowTests.class) annotation.

My BaseTest class is annotated with the following annotation @RunWith(MyJUnitRunner.class).

I've set up a Gradle task that is expected to run only SlowTests. Here's my Gradle task:

task integrationTests(type: Test) {
    minHeapSize = "768m"
    maxHeapSize = "1024m"
    testLogging {
        events "passed", "skipped", "failed"
        outputs.upToDateWhen {false}
    }
    reports.junitXml.destination = "$buildDir/test-result"
    useJUnit {
        includeCategories 'testutils.SlowTests'
    }
}

When I run the task, my tests aren't run. I've pinpointed this issue to be related to the custom runner MyJUnitRunner on the BaseTest. How can I set up my Gradle or test structure so that I can use a custom runner while using the Suite.

Answer 1

The solution to this turned out to smaller and trickier than I thought. Gradle was using my custom test runner and correctly invoking the filter method. However, my runner reloads all test classes through its own classloader for Javaassist enhancements.

This lead to the issue that SlowTest annotation was loaded through the Gradle classloader but when passed to my custom runner, the runner checked if the class was annotated with that annotation. This check never resolved correctly as the equality of the SlowTest annotation loaded through two different classloaders was different.

--

Since I've already done the research, I'll just leave this here. After days of digging through the Gradle and the (cryptic) JUnit sources, here's what I got.

Gradle simply doesn't handle any advanced JUnit functionality except the test categorization. When you create a Gradle task with the include-categories or the exclude-categories conditions, it builds a CategoryFilter. If you don't know, a Filter is what JUnit gives to the test-runner to decide whether a test or a test method should be filtered out. The test runner must implement the Filterable interface.

JUnit comes with multiple runners, the Categories is just another one of them. It extends a family of test runners called Suite. These suite based runners are designed to run a "suite" of tests. A suite of tests could be built by annotation introspection, by explicitly defining tests in a suite or any other method that builds a suite of tests.

In the case of the Categories runner, JUnit has it's own CategoryFilter but Gradle doesn't use that, it uses it's own CategoryFilter. Both provide more or less the same functionality and are JUnit filters so that can be used by any suite that implements Filterable.

The actual class in the Gradle responsible for running the JUnit tests is called JUnitTestClassExecuter. Once it has parsed the command line options it requests JUnit to check the runner should be used for a test. This method is invoked for every test as seen here.

The rest is simply up to JUnit. Gradle just created a custom RunNotifier to generate the standard XML files representing test results.

I hope someone finds this useful and saved themselves countless hours of debugging.

TLDR: You can use any runner in Gradle. Gradle has no specifics pertaining to runners. It is JUnit that decided the runners. If you'd like to know what runner will be used for your test, you can debug this by calling Request.aClass(testClass).getRunner(). Hack this somewhere into your codebase and print it to the console. (I wasn't very successful in attaching a debugger to Gradle.)