Gradle: optimize running tests in parallel

Question

I am experimenting with Gradle's capability for running tests in parallel. The main setting I have found is the maxParallelForks property of Test tasks. I expected the behavior of that setting to be similar to having a Executors.newFixedThreadPool to execute the tests. Namely, a fixed number of threads (processes in the case of Gradle) are executing concurrently; whenever one thread finishes the work, a new one is activated in the pool.

However, the behavior of Gradle is fundamentally different in a less optimal way. It looks like that Gradle divides the test classes into a number equal to maxParallelForks of groups, and then Gradle spawns a process for each group and let those processes execute in parallel. The problem with this strategy is obvious: it cannot dynamically adjust the execution based on the time needed by a test class.

For example, suppose you have 5 classes and maxParallelForks is set to 2. Among the five classes, there is a slow one and the rest are relatively quick. An ideal strategy would be let one process execute the slow one and the other process the quick ones. However, what Gradle does is group the slow one together with one or two quick ones and spawn two processes to execute two groups of classes, which is certainly less optimal than the ideal case.

Here is a simple demo.

A slow class:

class DemoTest {
    @Test
    void one() {
        Thread.sleep( 5000 )
        println System.getProperty('org.gradle.test.worker') + ": " + new Date().format('HH:mm:ss')
        assert 1 == 1
    }

    @Test
    void two() {
        Thread.sleep( 5000 )
        println System.getProperty('org.gradle.test.worker') + ": " + new Date().format('HH:mm:ss')
        assert 1 == 1
    }
}

Quick classes (DemoTest2-4, with identical class body):

class DemoTest2 {
    @Test
    void one() {
        Thread.sleep( 1000 )
        println System.getProperty('org.gradle.test.worker') + ": " + new Date().format('HH:mm:ss')
        assert 1 == 1
    }

    @Test
    void two() {
        Thread.sleep( 1000 )
        println System.getProperty('org.gradle.test.worker') + ": " + new Date().format('HH:mm:ss')
        assert 1 == 1
    }
}

All the classes are in package junit, which happens to be the same name as a famous test framework :-)

Here is a possible output:

junit.DemoTest2 > one STANDARD_OUT
    2: 14:54:00

junit.DemoTest2 > two STANDARD_OUT
    2: 14:54:01

junit.DemoTest4 > one STANDARD_OUT
    2: 14:54:02

junit.DemoTest4 > two STANDARD_OUT
    2: 14:54:03

junit.DemoTest > one STANDARD_OUT
    3: 14:54:04

junit.DemoTest > two STANDARD_OUT
    3: 14:54:09

junit.DemoTest3 > one STANDARD_OUT
    3: 14:54:10

junit.DemoTest3 > two STANDARD_OUT
    3: 14:54:11

junit.DemoTest5 > one STANDARD_OUT
    3: 14:54:12

junit.DemoTest5 > two STANDARD_OUT
    3: 14:54:13

As you can see, the slow class DemoTest is grouped with two quick classes. The total run time is about 13 seconds, which could have been 10 seconds, if the quick classes were grouped together.

So, is there any straightforward way to optimize this behavior in Gradle without resorting to a custom JUnit runner?

Thank you very much.

Answer 1

This can only be optimized by making changes to the Gradle codebase.