JUnit4 skip test(s) according to custom java annotations

Question

I would like my JUnit4 Tests to be executed according to a custom annotation that I created with Java. The purpose of this custom annotation is for JUnit4 to note that the test should only be run if the machine's platform matches the one specified in the annotation.

Say I have the following annotation:

public @interface Annotations {
    String OS();
    ...
}

And the following Tests:

public class myTests{

    @BeforeClass
    public setUp() { ... }

    @Annotations(OS="mac")
    @Test
    public myTest1() { ... }

    @Annotations(OS="windows")
    @Test
    public myTest2() { ... }

    @Annotation(OS="unix")
    @Test
    public myTest3() { ... }

}

If I were to execute these tests in a Mac machine, then only myTest1() should be executed and the rest should be ignored. However, I am currently stuck on how I should implement this. How do I let JUnit read my custom annotation and check whether the test should be run or not.

Answer 1

The best way I found on having exactly this behavior and having them as skipped tests for awareness in the report is using your own runner (like in the answer of AlexR) but overriding the runChild method which allows the test to be picked but handled like an ignore and not completely excluded.

The annotation to be used

@Retention(RetentionPolicy.RUNTIME)
public @interface TargetOS {
    String family();

    String name() default "";

    String arch() default "";

    String version() default "";
}

The JUnit runner

public class OSSensitiveRunner extends BlockJUnit4ClassRunner {
    public OSSensitiveRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    @Override
    protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
        Description description = describeChild(method);
        if (method.getAnnotation(Ignore.class) != null) {
            notifier.fireTestIgnored(description);
        } else if (method.getAnnotation(TargetOS.class) != null) {
            final TargetOS tos = method.getAnnotation(TargetOS.class);
            String name = tos.name().equals("") ? null : tos.name();
            String arch = tos.arch().equals("") ? null : tos.arch();
            String version = tos.version().equals("") ? null : tos.version();
            if (OS.isOs(tos.family(), name, arch, version)) {
                runLeaf(methodBlock(method), description, notifier);
            } else {
                notifier.fireTestIgnored(description);
            }
        } else {
            runLeaf(methodBlock(method), description, notifier);
        }
    }
}

Usage in a test

@RunWith(OSSensitiveRunner.class)
public class SeleniumDownloadHelperTest {
...

And restricting a specific method

@Test
@TargetOS(family = "windows")
public void testGetFileFromUrlInternetExplorer() throws Exception {
    ...
}

Answer 2

You can either use categories, or you can implement your own custom JUnit runner. Extending the default JUnit runner is pretty straightforward, and allows you to define the list of tests to be run in any way you might want. This includes looking for only those test methods with a specific annotation. I am including code samples below, you can use them as a basis for your own implementation:

Annotation:

@Retention(RetentionPolicy.RUNTIME)
public @interface MyCustomAnnotation {
   String OS();
}

Custom Runner Class:

public class MyCustomTestRunner extends BlockJUnit4ClassRunner {

   public MyCustomTestRunner(final Class<?> klass) throws InitializationError {
      super(klass);
   }

   @Override
   protected List<FrameworkMethod> computeTestMethods() {
      // First, get the base list of tests
      final List<FrameworkMethod> allMethods = getTestClass()
            .getAnnotatedMethods(Test.class);
      if (allMethods == null || allMethods.size() == 0)
         return allMethods;

      // Filter the list down
      final List<FrameworkMethod> filteredMethods = new ArrayList<FrameworkMethod>(
            allMethods.size());
      for (final FrameworkMethod method : allMethods) {
         final MyCustomAnnotation customAnnotation = method
               .getAnnotation(MyCustomAnnotation.class);
         if (customAnnotation != null) {
            // Add to accepted test methods, if matching criteria met
            // For example `if(currentOs.equals(customAnnotation.OS()))`
            filteredMethods.add(method);
         } else {
            // If test method doesnt have the custom annotation, either add it to
            // the accepted methods, or not, depending on what the 'default' behavior
            // should be
            filteredMethods.add(method);
         }
      }

      return filteredMethods;
   }
}

Sample Test Class:

@RunWith(MyCustomTestRunner.class)
public class MyCustomTest {
   public MyCustomTest() {
      super();
   }

   @Test
   @MyCustomAnnotation(OS = "Mac")
   public void testCustomViaAnnotation() {
      return;
   }
}

Answer 3

This is exactly what JUnit categories (see this short introduction) are made for.

After you marked all the tests with the appropriate category (using @Category), you can than create suites that run all the tests but those of the wrong category or all the tests that have the right category. (using the @IncludeCategory and @ExcludeCategory, you can combine them to narrow down your selection)

Categories can be used on Suite, Test-class and even Test-Method level.

Here are more informations regarding JUnit Categories