What is limiting number of active workers in a Gradle build?

Question

I am trying to run tests in parallel on my laptop, which has 4 pyhsical and 8 logical CPUs.

➜  sysctl -n hw.ncpu
8
➜  sysctl -n hw.physicalcpu
4

I would like to run at most 4 tests in parallel.
This is the command that I am running:

./gradlew remoteChromeTest -Pparallel=4 --continue --max-workers=8

The parallel property goes into maxParallelForks of the Test task.

Still, Gradle only seems to occupy at most 4 workers concurrently (one of which seems to be reserved for the build itself), which coincides with the number of physical CPUs (not sure if that matters).

The output looks like this:

> Task :remoteChromeTest
Running tests in parallel using 4 processes.
<============-> 92% EXECUTING [12s]
> :remoteChromeTest > 0 tests completed
> :remoteChromeTest > Executing test spec.Spec1
> IDLE
> :remoteChromeTest > Executing test spec.Spec2
> :remoteChromeTest > Executing test spec.Spec3

So, only 3 tests are running in parallel.

What am I missing here? The documentation suggests this could be cranked up beyond the number of actual CPUs. The weird part is that one of the workers shows up as IDLE.

I am getting the same behaviour on another machine, which has only 2 physical cores, and there it is limited to two processes, i.e. no parallel execution at all.

Why is one worker IDLE?

Answer 1

--max-workers is not only for tests execution but also for parallel project execution. In the example, however, you are limiting tests execution parallelism to 4. Try setting maxParallelForks to more than 4 and you will see more parallelism in tests execution. But anyway it will be limited to the minimum between the number of tests and --max-workers.

You can also enable parallel project execution by using the --parallel option. And again, the build parallelism, including the test parallelism, will be limited to --max-workers:

subprojects {
    tasks.withType(Test) {
        maxParallelForks = 8
    }
}

gradle clean build --parallel --max-workers=6
...
<<===========--> 85% EXECUTING [15s]
>> :module-1:test > 4 tests completed
>> :module-2:test > 2 tests completed
>> :module-2:test > Executing test so.Module2Spec3
>> :module-2:test > Executing test so.Module2Spec1
>> :module-1:test > Executing test so.Module1Spec4
>> :module-1:test > Executing test so.Module1Spec2
>> :module-1:test > Executing test so.Module1Spec1
>> :module-1:test > Executing test so.Module1Spec3

gradle clean build --parallel --max-workers=12
...
<===========--> 85% EXECUTING [13s]
> :module-1:test > 0 tests completed
> :module-2:test > 0 tests completed
> :module-2:test > Executing test so.Module2Spec1
> :module-2:test > Executing test so.Module2Spec2
> :module-2:test > Executing test so.Module2Spec3
> :module-2:test > Executing test so.Module2Spec4
> :module-1:test > Executing test so.Module1Spec1
> :module-1:test > Executing test so.Module1Test2
> :module-1:test > Executing test so.Module1Spec4
> :module-1:test > Executing test so.Module1Test3
> :module-1:test > Executing test so.Module1Test1
> :module-1:test > Executing test so.Module1Spec3
> :module-1:test > Executing test so.Module1Spec2
> :module-1:test > Executing test so.Module1Test4