JUnit and Surefire Parallel Tests - ForkCount & ThreadCount

Question

I'm running Selenium tests on the Selenium Grid using the Surefire Plugin to execute tests. In terms of my test breakdown I have several classes, some of which have 1 test in there and some more than one test.

So on my Grid i have 30 chrome web drivers and I want to execute all tests within all classes in parallel.

I've read how to do this using the parallel parameter which i have set as:

            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.17</version>
                <configuration>
                    <includes>
                        <include>${testSuite}</include>
                    </includes>
                    <parallel>all</parallel>
                    <useSystemClassLoader>false</useSystemClassLoader>
                    <perCoreThreadCount>false</perCoreThreadCount>
                    <threadCount>20</threadCount>
                    <browser>${browser_type}</browser>
                </configuration>
            </plugin>

However this doesnt seem to fill all the Chrome web drivers I have available.

If i then use forkCount, like:

<forkCount>20</forkCount>
<reuseForks>true</reuseForks>

Then when the test execution first starts, all web drivers are filled however it quickly starts dropping and behaving one at a time.

So my questions:

• Is there a relationship between forkCount and threadCount
• Is there anything additional I need to do to really get this running in parallel?

Thanks.

Answer 1

There are so many configuration for running test in parallel.

According to the documentation:

forkCount

The parameter forkCount defines the maximum number of JVM processes that Surefire will spawn concurrently to execute the tests. It supports the same syntax as -T in maven-core: if you terminate the value with a C, that value will be multiplied with the number of available CPU cores in your system. For example forkCount=2.5C on a Quad-Core system will result in forking up to ten concurrent JVM processes that execute tests.

...

The default setting is forkCount=1/reuseForks=true, which means that Surefire creates one new JVM process to execute all tests in one maven module.

threadCount

When using reuseForks=true and a forkCount value larger than one, test classes are handed over to the forked process one-by-one. Thus, parallel=classes would not change anything. However, you can use parallel=methods: classes are executed in forkCount concurrent processes, each of the processes can then use threadCount threads to execute the methods of one class in parallel.

As fair as I understand, threadCount is like a sub threading for each fork.

You can tweak these params to improve your test performance, for instance you can have:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.17</version>
    <configuration>
         <includes>
              <include>${testSuite}</include>
         </includes>
         <parallel>all</parallel>
         <useSystemClassLoader>false</useSystemClassLoader>
         <perCoreThreadCount>false</perCoreThreadCount>
         <forkCount>2.0C</forkCount>
         <reuseForks>true</reuseForks>
         <threadCount>20</threadCount>
         <browser>${browser_type}</browser>
    </configuration>
</plugin>

You can find more details in its site:

https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

Answer 2

You have to provide explicit junit test provider:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.18.1</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>2.18.1</version>
        </dependency>
    </dependencies>
    <configuration>
        <parallel>all</parallel>
        <useUnlimitedThreads>true</useUnlimitedThreads>
        <useSystemClassLoader>false</useSystemClassLoader>
        <includes>
            <include>${testSuite}</include>
        </includes>
        <systemPropertyVariables>
            <browser>${browser_type}</browser>
        </systemPropertyVariables>
     </configuration>
</plugin>

And you should use JUnit 4.7+ as older versions does not work with parallel testing correctly.

Also you can omit fork-related parameters if your tests do NOT affect JVM runtime (usually it's not the case).

Or migrate your tests to TestNG - it is more elegant framework and it works with parallel testing much better, then JUnit (imo).