Howto have maven surefire execute JUnit and TestNG test properly?

Question

Since a while it is possible to configure maven surefire to execute jUnit tests and testNG tests in one build. I won't expand on the reasons why I'm doing that (ok, hint: testNG is our standard framework but some frameworks like jnario require jUnit).

The general way to do it is to configure the surefire plugin like this:

<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${surefire.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>${surefire.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-testng</artifactId>
            <version>${surefire.version}</version>
        </dependency>
    </dependencies>
</plugin>

(taken from here)

This works quite well, jUnit tests and testNG tests get executed.

BUT - now I see that testNG tries to execute the jUnit tests too (and maybe vice-versa) - with no success, of course, because it won't see any of its annotations, but it looks like it re-marks the tests to "passed"... anyway, some reporting tools don't show test fails in jUnit tests anymore unless I comment the second dependency entry.

Is there any better way to configure surefire so that tests from both frameworks are executed ONLY by their test runners?

Answer 1

Had the same problem as you. Zero results in TestNG reports but looks like tests were runned. Finally get this work on two separate executions of maven-surefire-plugin

1. First remove the dependencies section from plugin.
   • otherwise if you use <testNGArtifactName> (and we'll need it) tests will be executed by TestNG anyway and will fail on java.lang.NullPointerException
2. Have some separation of TestNG tests and JUnit tests. In our case NG test class names ends with TestNg
3. skip default execution
4. define two <executions> like below:

<plugin>
    <!-- configured to work with JUnit and TestNg in same project separatelly -->
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>              
    <configuration>
        <!-- whatever you need... -->
        <!-- skip the default execution, we have separate for Ng and for JUnit -->
        <skip>true</skip>
    </configuration>
    <executions>
        <execution>
            <id>TestNg-execution</id>
            <phase>test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>
                <!-- overwrite skip from default config -->
                <skip>false</skip>
                <includes>
                    <include>%regex[.*TestNg.*]</include>
                </includes>
                <!-- used to skip JUnit profider -->
                <junitArtifactName>dev:null</junitArtifactName>
                <!-- to continue on next execution in case of failures here -->
                <testFailureIgnore>true</testFailureIgnore>
            </configuration>
        </execution>
        <execution>
            <id>JUnit-execution</id>
            <phase>test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>

                <skip>false</skip>
                <!-- used to skip TestNg profider -->
                <testNGArtifactName>dev:null</testNGArtifactName>
                <excludes>
                    <exclude>%regex[.*TestNg.*]</exclude>                               
                </excludes>
            </configuration>                    
        </execution>
    </executions>
</plugin>

Tested with:

<junit-version>4.11</junit-version>
<maven-surefire-plugin-version>2.16</maven-surefire-plugin-version>
<org.testng-version>6.8.7</org.testng-version>

This configuration is the marge of ideas from:
Running TestNG-, JUnit3- and JUnit4-tests with Maven Surefire in one run,
your SUREFIRE bug report,
surefire plugin configuration

Hope it helps.