Combining profiles in Maven to skip multiple categories of unit tests?

Question

I am having a problem trying to configure Maven to excluded some categories of unit tests based on multiple profiles.

I have two categories defined for unit testing: SlowTest for those unit tests that take a very long time to run, and WindowsTest for those unit tests that can only be executed within a Windows environment.

I have two profiles defined in my project's pom.xml file as follows:

<profiles>
  <!-- Exclude any tests in `SlowTest` category when profile 'skipSlowTests' is specified. -->
  <profile>
    <id>skipSlowTests</id>
    <activation>
      <activeByDefault>false</activeByDefault>
    </activation>
    <build>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <configuration>
            <excludedGroups>SlowTest.class</excludedGroups>
          </configuration>
        </plugin>
      </plugins>
    </build>
  </profile>  

  <!-- Skip any tests in 'WindowsTest' category when not running on Windows. -->
  <profile>
    <id>skipWindowsTests</id>
    <activation>
      <os><family>!windows</family></os>
    </activation>
    <build>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <configuration>
            <excludedGroups>WindowsTest.class</excludedGroups>
          </configuration>
        </plugin>
      </plugins>
    </build>
  </profile>  

</profiles>

So for example when I want to run the tests and exclude any slow tests, I can execute mvn -PskipSlowTests test. And If I want to skip any Windows tests I can execute mvn test on a non-Windows OS (or explicitly specify -PskipWindowsTests on the command line).

The problem here is that when both the skipSlowTests and skipWindowsTests profiles are activated then only the Windows tests are skipped. The slow tests still run. When maven constructs the effective pom.xml file, it applies the skipSlowTests profile and configures maven-surefire-plugin to exclude the groups SlowTest.class. Then the skipWindowsTests profile is applied and this overwrites the excluded groups to be WindowsTest.class.

What I really want is the excluded groups to be SlowTest.class,WindowsTest.class when both profiles are activated, but I cannot figure out a way to do this.

There's no way that I can see in maven of appending a value to a property, e.g.

<properties>
  <excludedGroups>IgnoreTest.class</excludedGroups>
</properties>

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <excludedGroups>${excludedGroups}</excludedGroups>
  </configuration>
</plugin>

<profile>
  <id>skipSlowTests</id>
  <property>
    <excludedGroups>${excludedGroups},SlowTest.class</excludedGroups>
  </property>
  ...
</profile>

<profile>
  <i>skipWindowsTests</id>
  <property>
    <excludedGroups>${excludedGroups},WindowsTest.class</excludedGroups>
  </property>
  ...
</profile>

This pom.xml is broken, because there's recursion in assigning a value to the excludedGroups property.

And there's no way to set up a third profile (e.g. skipSlowTestsAndWindowsTests) that activates whenever both the skipSlowTests and skipWindowsTests are activated.

Any suggestions here? Maybe adding separate phases to maven-surefire-plugin (one for slow tests, one for windows tests, one for all others) and using profiles to determine which phases run?

Edit #1

I tried creating separate execution phases for the maven-surefire-plugin as follows:

<!-- By default, do not skip slow tests or windows tests -->
<properties>
  <skipSlowTests>false</skipSlowTests>
  <skipWindowsTests>false</skipWindowsTests>
</properties>

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <executions>
    <!-- Default execution will skip SlowTest and WindowsTest categories -->
    <execution>
      <id>default-test</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <excludedGroups>SlowTest,WindowsTest</excludedGroups>
      </configuration>
    </execution>
    <!-- Slow Test execution will run the slow tests only (if not skipped) -->
    <execution>
      <id>slow-test</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <skip>${skipSlowTests}</skip>
        <groups>SlowTest</groups>
      </configuration>
    </execution>
    <!-- Windows Test execution will run the windows tests only (if not skipped) -->
    <execution>
      <id>windows-test</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <skip>${skipWindowsTests}</skip>
        <groups>WindowsTest</groups>
      </configuration>
    </execution>
  </executions>
</plugin>

<profiles>
  <!-- Skip slow tests when skipSlowTests profile enabled -->
  <profile>
    <id>skipSlowTests</id>
    <properties>
      <skipSlowTests>true</skipSlowTests>
    </properties>
  </profile>
  <!-- Skip windows tests when skipWindowsTests profile enabled -->
  <profile>
    <id>skipWindowsTests</id>
    <properties>
      <skipWindowsTests>true</skipWindowsTests>
    </properties>
    ...
  </profile>
</profiles>

This configuration will run tests in three separate executions: the default-test execution will run every test except for slow tests and windows tests; the slow-test execution will run the slow tests only; and the windows-test execution will run the windows tests only. And if both the skipSlowTests and skipWindowsTests profiles are enabled, then both slow tests and windows tests will be skipped.

The problem here is when there is a test marked with both SlowTest and WindowsTest categories:

@Test @Category({SlowTest.class,WindowsTest.class})
public void slowWindowsTestCase() { ... }

When the skipSlowTests profile only is enabled, then the windows-test execution is still run, and since the slowWindowsTestCase method is marked as a windows test, it will be executed, even though we want to skip slow tests. Similarly, this test case will also be executed when the skipWindowsTests profile only is enabled. The only way to skip this test case is to specify both the skipSlowTests and skipWindowsTests profiles.

I tried modifying the executions as follows:

    <execution>
      <id>slow-test</id>
        ... 
        <groups>SlowTest</groups>
        <excludedGroups>WindowsTest</excludedGroups>
        ...
    </execution>
    <execution>
      <id>windows-test</id>
        ... 
        <groups>WindowsTest</groups>
        <excludedGroups>SlowTest</excludedGroups>
        ...
    </execution>

But now the slowWindowsTestCase method is never executed.

Edit #2

And even if I could get the test executions working correctly, I would need to add a dependency on maven-surefire-plugin so that the SlowTest and WindowsTest interfaces can be loaded during testing. I have put maven-surefire-plugin in the pluginManagement section of my parent POM, but any dependency defined for this plugin cannot have test scope. As a result, running a test outside of the module that defines the SlowTest and WindowsTest interfaces will not work. I can add a test scope dependency for the plugin in the profiles, but that will not cover the case when no profiles are active.

Answer 1

You can make the default profile exclude all groups, and for each profile, you add one execution of the failsafe/surefire plugin including only one group.

Answer 2

What about having each profile define a property excludedGroups and concatenate those into one property at the very start of POM execution?

<build>
  ...
  <plugin>
    <groupId>org.codehaus.gmavenplus</groupId>
    <artifactId>gmavenplus-plugin</artifactId>
    <version>1.6</version>
    <dependencies>
      <dependency>
        <groupId>org.codehaus.groovy</groupId>
        <artifactId>groovy-all</artifactId>
        <version>2.4.9</version>
        <scope>runtime</scope>
      </dependency>
    </dependencies>
    <executions>
      <execution>
        <id>excludedGroups</id>
        <phase>initialize</phase>
        <goals>
          <goal>execute</goal>
        </goals>
        <configuration>
          <scripts>
            <script>
              <![CDATA[
              def groups = ""
              (project.activeProfiles).each{ profile -> groups += profile.properties.excludedGroups + "," }
              project.properties.excludedGroups = groups.substring(0, groups.length() - 1)
              ]]>
            </script>
          </scripts>
        </configuration>
      </execution>
    </executions>
  </plugin>
</build>

<profiles>
  <!-- Exclude any tests in `SlowTest` category when profile 'skipSlowTests' is specified. -->
  <profile>
    <id>skipSlowTests</id>
    <properties>
      <excludedGroups>SlowTest.class</activeByDefault>
    </properties>
  </profile>  

  <!-- Skip any tests in 'WindowsTest' category when not running on Windows. -->
  <profile>
    <id>skipWindowsTests</id>
    <activation>
      <os><family>!windows</family></os>
    </activation>
    <properties>
      <excludedGroups>WindowsTest.class</excludedGroups>
    </properties>
  </profile>
</profiles>

Now you can use ${excludedGroups} as value for the surefire plugin configuration.