How to group/categorize large number of JUnit tests

Question

In our project, we currently have a large number of (junit) tests that are split into three categories: unit, integration, wicket.

I now want to group these tests so I can run only one (or two) of those categories. The only thing I found are junit test suites and categories as described here: http://www.wakaleo.com/component/content/article/267

My problem is, I don't want to declare every single test in the Test Suits with @SuiteClasses.

Is there a way to add the suite classes with wildcards / patterns?

Answer 1

Assuming my understanding of the question is correct, it actually can be done using JUnit. The code below was used with JUnit 4.11 and allowed us to split all tests into 2 categories: "uncategorized" and Integration.

IntegrationTestSuite.java

/**
 * A custom JUnit runner that executes all tests from the classpath that
 * match the <code>ca.vtesc.portfolio.*Test</code> pattern 
 * and marked with <code>@Category(IntegrationTestCategory.class)</code>
 * annotation. 
 */
@RunWith(Categories.class)
@IncludeCategory(IntegrationTestCategory.class)
@Suite.SuiteClasses( { IntegrationTests.class })
public class IntegrationTestSuite {
}

@RunWith(ClasspathSuite.class)
@ClasspathSuite.ClassnameFilters({ "ca.vtesc.portfolio.*Test" })
class IntegrationTests {
}

UnitTestSuite.java

/**
  * A custom JUnit runner that executes all tests from the classpath that match
  * <code>ca.vtesc.portfolio.*Test</code> pattern.
  * <p>
  * Classes and methods that are annotated with the
  * <code>@Category(IntegrationTestCategory.class)</code> category are 
  * <strong>excluded</strong>.
  */

@RunWith(Categories.class)
@ExcludeCategory(IntegrationTestCategory.class)
@Suite.SuiteClasses( { UnitTests.class })
public class UnitTestSuite {
}

@RunWith(ClasspathSuite.class)
@ClasspathSuite.ClassnameFilters({ "ca.vtesc.portfolio.*Test" })
class UnitTests {
}

IntegrationTestCategory.java

/**
 * A marker interface for running integration tests.
 */
public interface IntegrationTestCategory {
}

The first sample test below is not annotated with any category so all its test methods will be included when running the UnitTestSuite and excluded when running IntegrationTestSuite.

public class OptionsServiceImplTest {
    @Test
    public void testOptionAssignment() {
        // actual test code
    }
}

Next sample is marked as Integration test on the class level which means both its test methods will be excluded when running the UnitTestSuite and included into IntegrationTestSuite:

@Category(IntegrationTestCategory.class)
public class PortfolioServiceImplTest {
    @Test
    public void testTransfer() {
        // actual test code
    }
    @Test
    public void testQuote() {
    }
}

And the third sample demos a test class with one method not annotated and the other marked with the Integration category.

public class MarginServiceImplTest {
    @Test
    public void testPayment() {
    }
    @Test
    @Category(IntegrationTestCategory.class)
    public void testCall() {
    }
}

Answer 2

Try using ClasspathSuite

I also had the same problem where I had more then 5500 jUnit tests. I categorised then into 3 groups and created 3 suites using the above jUnit extension. Its great.

Answer 3

You could put them in different packages. Most IDEs have a way to run all the tests in a given package. It's also pretty simple to find all the test classes in a package with a shell script for running tests as part of a build or whatever. I don't know if there's a way to do it with ant, but I would imagine so.

TestNG lets you tag tests as being in particular groups, then run those groups. That sound like exactly what you want, apart from the fact that it's not JUnit!

You could abuse JUnit's assumption mechanism to do what you want: have a system property for each group of tests, and then start each test by assuming that the appropriate property is set. Running all tests will run everything, but everything you don't want will be ignored.