Run Gradle tests with multiple Java toolchains

Question

I've got a Gradle project which uses a Java version specified with the toolchain API:

val minimumJava = JavaLanguageVersion.of(8)
val maximumJava = JavaLanguageVersion.of(16)
java {
    toolchain {
        languageVersion.set(minimumJava)
        vendor.set(JvmVendorSpec.ADOPTOPENJDK)
    }
}

I would like to be able to compile with the minimum supported Java version, then run the tests with all the JDKs the project supports. I tried the following, but apparently only the original tests get executed, all other tests don't, even though the required JDKs get correctly downloaded and set up:

for (javaVersion in JavaLanguageVersion.of(minimumJava.asInt() + 1)..maximumJava) {
    val base = tasks.test.get()
    val testTask = tasks.register<Test>("testUnderJava${javaVersion.asInt()}") {
        javaLauncher.set(
            javaToolchains.launcherFor {
                languageVersion.set(javaVersion)
            }
        )
        classpath = base.classpath
        testClassesDirs = base.testClassesDirs
        isScanForTestClasses = true
    }
    tasks.test.configure { finalizedBy(testTask) }
}

Here is a run in a dumb terminal:

❯ TERM=dumb ./gradlew test testUnderJava10 --rerun-tasks --scan
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on

<<<SNIP>>>

> Task :testClasses

> Task :test
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on

Gradle Test Executor 4 STANDARD_OUT
    ~~~ Kotest Configuration ~~~
    -> Parallelization factor: 1
    -> Concurrent specs: null
    -> Global concurrent tests: 1
    -> Dispatcher affinity: true
    -> Default test timeout: 600000ms
    -> Default test order: Sequential
    -> Default isolation mode: SingleInstance
    -> Global soft assertions: false
    -> Write spec failure file: false
    -> Fail on ignored tests: false
    -> Spec execution order: SpecExecutionOrder
    -> Remove test name whitespace: false
    -> Append tags to test names: false
    -> Extensions
      - io.kotest.engine.extensions.SystemPropertyTagExtension
      - io.kotest.core.extensions.RuntimeTagExtension
      - io.kotest.engine.extensions.RuntimeTagExpressionExtension


org.danilopianini.template.test.Tests > A greeting should get printed STARTED

org.danilopianini.template.test.Tests > A greeting should get printed STANDARD_OUT
    [:hello=SUCCESS]

    > Task :hello
    Hello from Danilo Pianini

    BUILD SUCCESSFUL in 2s
    1 actionable task: 1 executed


org.danilopianini.template.test.Tests > A greeting should get printed PASSED

<<<Other tests have no output!>>>

> Task :testUnderJava9
> Task :testUnderJava8
> Task :testUnderJava16
> Task :testUnderJava15
> Task :testUnderJava14
> Task :testUnderJava13
> Task :testUnderJava12
> Task :testUnderJava11
> Task :testUnderJava10

BUILD SUCCESSFUL in 23s
36 actionable tasks: 36 executed

<<<SNIP>>>

From the build scan, it appears that tests are not executed but those with JDK8. I'm puzzled, the docs say that this should be straightforward:

tasks.register<Test>("testsOn14") {
    javaLauncher.set(javaToolchains.launcherFor {
        languageVersion.set(JavaLanguageVersion.of(14))
    })
}

Answer 1

I think I worked out the root cause of the issues I was experiencing, I'm posting the solution in case someone else runs into similar issues. I had the following tests configuration:

tasks.test {
    useJUnitPlatform()
    testLogging {
        showStandardStreams = true
        showCauses = true
        showStackTraces = true
        events(*org.gradle.api.tasks.testing.logging.TestLogEvent.values())
        exceptionFormat = org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
    }
}

Which was instructing the task called test to useJunitPlatform(). This setting does not get automatically propagated to all subsequent Test tasks (of course). So, in this case, the solution is simply to use instead:

tasks.withType<Test> { 
    // Same configuration as above
}

Update 2022-03-16

I decided to create a multi-JVM testing plugin for Gradle, so that all the test tasks get created and much less boilerplate is required across projects.

Update 2023-01-17

Gradle recommends using the task configuration avoidance API.

tasks.withType<Test>().configureEach { 
    // Same configuration as above
}