Parallel Test Execution with Gradle maxParallelForks property

Question

I have been searching for the answers last few days and unable to find one. The closest answer I could find is this which does not exactly answer the questions I have.

By the way, I have a Selenium Test Project which is based on Gradle. We build the project on Jenkins run the tests in 20 concurrent threads. Total number of unique test classes I have is 87. So, I expect gradle to exeute at least 5 batches The test project is build using Cucumber JVM, build and triggers tests by Jenkins to Selenium Hub. I tried to increase the parallelism of the tests by utilizing the grid as much as possible. But, the problem started when the number of tests started growing.

When I started the tests from Jenkins, I observed at first shot the test executed all 20 test processes and I see the second batch also started with same amount of processes. After the second batch the processes went back to single mode and the entire job took 14 hours to complete which defeats the purpose of having parallel test execution.

Gradle properties:

jvmArgs '-Xms128m', '-Xmx1024m', '-XX:MaxPermSize=128m'
Runtime.runtime.availableProcessors().toString()) as int        
maxParallelForks =  PropertyUtils.getProperty('test.parallel', '15') as int
forkEvery =  PropertyUtils.getProperty('test.forkEvery', '0') as int

CLI:

gradle clean test -Dtest.single=*TestRun --info

I have read all the documents I can possibly find but failed to get answer. It would be greatly appreciated if someone can help me with these questions

1. How Gradle batch the test runner internally? For example if 20 executors starts and test 1,2,3 done executing faster than the others, do the three executors gets three more test classes or waits for the entire batch to finish executing?

2. Can forkEvery impact how the execution works during parallel testing?

Jenkins log

Successfully started process 'Gradle Test Executor 6'

Successfully started process 'Gradle Test Executor 13'

Successfully started process 'Gradle Test Executor 14'

Successfully started process 'Gradle Test Executor 5'

Successfully started process 'Gradle Test Executor 16'

Successfully started process 'Gradle Test Executor 8'

Successfully started process 'Gradle Test Executor 19'

Successfully started process 'Gradle Test Executor 4'

Successfully started process 'Gradle Test Executor 2'

Successfully started process 'Gradle Test Executor 11'

Successfully started process 'Gradle Test Executor 10'

Successfully started process 'Gradle Test Executor 18'

Successfully started process 'Gradle Test Executor 1'

Successfully started process 'Gradle Test Executor 20'

Successfully started process 'Gradle Test Executor 7'

Successfully started process 'Gradle Test Executor 9'

Successfully started process 'Gradle Test Executor 3'

Successfully started process 'Gradle Test Executor 15'

Successfully started process 'Gradle Test Executor 17'

Successfully started process 'Gradle Test Executor 12'

Gradle Test Executor 13 started executing tests.

Gradle Test Executor 14 started executing tests.

Gradle Test Executor 6 started executing tests.

Gradle Test Executor 5 started executing tests.

Gradle Test Executor 16 started executing tests.

Gradle Test Executor 19 started executing tests.

Gradle Test Executor 8 started executing tests.

Gradle Test Executor 4 started executing tests.

Gradle Test Executor 2 started executing tests.

Gradle Test Executor 10 started executing tests.

Gradle Test Executor 11 started executing tests.

Gradle Test Executor 18 started executing tests.

Gradle Test Executor 1 started executing tests.

Gradle Test Executor 20 started executing tests.

Gradle Test Executor 7 started executing tests.

Gradle Test Executor 3 started executing tests.

Gradle Test Executor 9 started executing tests.

Gradle Test Executor 17 started executing tests.

Gradle Test Executor 15 started executing tests.

Gradle Test Executor 12 started executing tests.

Answer 1

The default of forkEvery is 0

According to the documentation forkEvery is

The maximum number of test classes to execute in a forked test process. The forked test process will be restarted when this limit is reached. The default value is 0 (no maximum).

So gradle (and probably junit) will fork by classes not tests within the class. It sounds like a few of the 87 test classes have long running tests or a large number of tests and they end up in one forked test process. I would consider setting forkEvery to 1. This will ensure that each test class is sent to a new fork. If there is still an issue you may need to find which test classes are taking the most time. Consider splitting these classes up into smaller groups of tests so the tests get spread over each jvm. If it is one test that takes forever consider redesigning it and possibly creating smaller tests from it.

I do not believe that gradle runs tests in batches. As a worker becomes available it takes a test class from the queue of remaining tests. You would really have to look at how JUnit works as I'm sure gradle is simply passing these configurations to JUnit.