Better or custom JUnit test filtering

Question

I'd like to implement a better system to filter tests to run under JUnit in Eclipse. I'd like the @Categories, but not have to specify the list of classes in a @SuiteClasses({}) annotation , since I think that reduces the value, increases the amount of manual maintenance we have to do.

Is there some way to hook into the process where Eclipse runs JUnit test, to be able to do custom filtering on all the classes with a @Test in it? E.g. a class like this:

    @CustomFilteredTesting 
    public class TheCustomFilteredTestRun {
        public boolean includeThisTestClass(Class<?> klass) {
            // e.g. test whether klass is in a package or subsystem 
            // or a subtype of some interface. 
        } 
    }

Any help appreciated, -j

Answer 1

Eclipse uses the default JUnit code to run the tests, so you only need to teach JUnit your new trick and Eclipse will be able to do it.

That out of the way: How can you split tests into suites without @SuiteClasses?

Tough. From my experience, there are several solutions:

1. Write a test that reads all suites and tries to find all tests and then makes sure that each test is in one suite. This test will let you know when you forgot to add a new test to the suites.

   Nice: No JUnit magic involved, will keep you safe if you forget something (like to add a @Category to a new test).

   Drawback: Needs quite some ugly File reading/code parsing/reflection mojo.

2. You can write your own test runner by extending BlockJUnit4ClassRunner. That would allow you to add your own, custom annotations to classify tests and run them.

3. Add assumptions. org.junit.Assume is a nice little beast to silently fail a test. The test won't run (or it will run until the first assumption fails) but it also won't log an error. It's a bit like a smart @Ignore

   Write setup code that determines which tests to run and put assumeThat() in your code.

I actually prefer to use suites because:

1. The "do I have all tests" test tells when I'm wrong
2. I have a single place where I can see which test goes where
3. I can easily write more tests to make sure tests aren't repeated in suites

Answer 2

Consider using ClassPathSuite: http://johanneslink.net/projects/cpsuite.jsp. We use it and define a Suite as follows:

@RunWith(ClasspathSuite.class)
@ExcludeBaseTypeFilter({ NotUnitTestable.class })
public class AllTests {
}

Exclusion is not possible using Annotations so we simply defined a marker interface (NotUnitTestable). We start our tests this way both in Eclipse and in our command line build using the JUnit ANT integration.