Get a count for the number of times a @Category appears in a suite of tests in JUnit

Question

I have developed a full suite of automated tests using Java, Selenium, Junit, Maven.

For each test, they have one or more @Category annotations describing what area of the software each test covers. For instance:

@Test
@Category({com.example.core.categories.Priority1.class,
           com.example.core.categories.Export.class,
           com.example.core.categories.MemberData.class})


@Test
@Category({com.example.core.categories.Priority1.class,
           com.example.core.categories.Import.class,
           com.example.core.categories.MemberData.class})


@Test
@Ignore
@Category({com.example.core.categories.Priority2.class,
           com.example.core.categories.Import.class,
           com.example.core.categories.MemberData.class})

What I'm trying to do is find a way to get a count of how many tests contain any given category. All the possible categories are filenames in the //com/example/core/categories folder as a source list.

I've tried building a shell script to do a word count, which seems to work okay, but I would think there would be something more "built-in" to deal with @Category.

My biggest issue is that even if I get the right count, it is very possible that one or more of the tests are marked @Ignore which should nullify that tests @Category's but without heavy use of flags and reading every file line-by-line in order it throws off the correct count.

Is there a good way to itemize @Category's that also factors in @Ignore?

Example output

| Category                                     | Count |
|----------------------------------------------|------:|
| com.example.core.categories.Export.class     | 1     |
| com.example.core.categories.Import.class     | 1     |
| com.example.core.categories.MemberData.class | 2     |
| com.example.core.categories.Priority1.class  | 2     |
| com.example.core.categories.Priority2.class  | 0     |
| com.example.core.categories.Priority3.class  | 0     |

Answer 1

Dynamic 'Tests by Category' Computer

(Recommended method)

I tried a way to perform this with a counter in abstract layer but it was painful, having to add source code at beginning of each Test methods.

At end, this is the source code I wrote to answer your needs; it is quite heavy (reflection ...), but it is the less intrusive with existing source code, and answers totally to your needs.

First, you must create a Testsuite (containing various others Suites, or directly all the Test classes you want), to ensure at end, that all Tests for which you want statistics, have been loaded.

In this Suite, you have to implement a "final Hook", called @AfterClass which will be called once for all, when the whole Test suite has been fully managed by JUnit.

This the the Test Suite implementation I wrote for you:

package misc.category;

import java.lang.annotation.Annotation;
import java.lang.reflect.Method;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Vector;
import java.util.concurrent.atomic.AtomicInteger;

import org.junit.AfterClass;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({ UnitTestWithCategory.class })
public class TestSuiteCountComputer {

    public static final String MAIN_TEST_PACKAGES = "misc.category";

    private static final Class<?>[] getClasses(final ClassLoader classLoader)
            throws NoSuchFieldException, SecurityException, IllegalArgumentException, IllegalAccessException {
        Class<?> CL_class = classLoader.getClass();
        while (CL_class != java.lang.ClassLoader.class) {
            CL_class = CL_class.getSuperclass();
        }
        java.lang.reflect.Field ClassLoader_classes_field = CL_class.getDeclaredField("classes");
        ClassLoader_classes_field.setAccessible(true);
        Vector<?> classVector = (Vector<?>) ClassLoader_classes_field.get(classLoader);

        Class<?>[] classes = new Class[classVector.size()]; // Creates an array to avoid concurrent modification
                                                            // exception.
        return classVector.toArray(classes);
    }

    // Registers the information.
    private static final void registerTest(Map<String, AtomicInteger> testByCategoryMap, String category) {
        AtomicInteger count;
        if (testByCategoryMap.containsKey(category)) {
            count = testByCategoryMap.get(category);
        } else {
            count = new AtomicInteger(0);
            testByCategoryMap.put(category, count);
        }

        count.incrementAndGet();
    }

    @AfterClass
    public static void tearDownAfterClass() throws Exception {
        Map<String, AtomicInteger> testByCategoryMap = new HashMap<>();

        ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        while (classLoader != null) {
            for (Class<?> classToCheck : getClasses(classLoader)) {
                String packageName = classToCheck.getPackage() != null ? classToCheck.getPackage().getName() : "";
                if (!packageName.startsWith(MAIN_TEST_PACKAGES))
                    continue;

                // For each methods of the class.
                for (Method method : classToCheck.getDeclaredMethods()) {
                    Class<?>[] categoryClassToRegister = null;
                    boolean ignored = false;
                    for (Annotation annotation : method.getAnnotations()) {
                        if (annotation instanceof org.junit.experimental.categories.Category) {
                            categoryClassToRegister = ((org.junit.experimental.categories.Category) annotation).value();
                        } else if (annotation instanceof org.junit.Ignore) {
                            ignored = true;

                        } else {
                            // Ignore this annotation.
                            continue;
                        }
                    }

                    if (ignored) {
                        // If you want to compute count of ignored test.
                        registerTest(testByCategoryMap, "(Ignored Tests)");
                    } else if (categoryClassToRegister != null) {
                        for (Class<?> categoryClass : categoryClassToRegister) {
                            registerTest(testByCategoryMap, categoryClass.getCanonicalName());
                        }
                    }

                }

            }
            classLoader = classLoader.getParent();
        }

        System.out.println("\nFinal Statistics:");
        System.out.println("Count of Tests\t\tCategory");
        for (Entry<String, AtomicInteger> info : testByCategoryMap.entrySet()) {
            System.out.println("\t" + info.getValue() + "\t\t" + info.getKey());
        }

    }

}

You can adapt to your needs, in particular the constant I created at beginning, to filter package to consider.

Then you have nothing more to do than you already do.

For instance, this is my tiny Test Class:

package misc.category;

import org.junit.Test;
import org.junit.experimental.categories.Category;

public class UnitTestWithCategory {

    @Category({CategoryA.class, CategoryB.class})
    @Test
    public final void Test() {
        System.out.println("In Test 1");
    }

    @Category(CategoryA.class)
    @Test
    public final void Test2() {
        System.out.println("In Test 2");
    }

}

In this case, the output is:

In Test 1
In Test 2

Final Statistics:
Count of Tests      Category
    1       misc.category.CategoryB
    2       misc.category.CategoryA

And with Test case containing @Ignore annotation:

package misc.category;

import org.junit.Ignore;
import org.junit.Test;
import org.junit.experimental.categories.Category;

public class UnitTestWithCategory {

    @Category({CategoryA.class, CategoryB.class})
    @Test
    public final void Test() {
        System.out.println("In Test 1");
    }

    @Category(CategoryA.class)
    @Test
    public final void Test2() {
        System.out.println("In Test 2");
    }

    @Category(CategoryA.class)
    @Ignore
    @Test
    public final void Test3() {
        System.out.println("In Test 3");
    }   
}

You get the output:

In Test 1
In Test 2

Final Statistics:
Count of Tests      Category
    1       (Ignored Tests)
    1       misc.category.CategoryB
    2       misc.category.CategoryA

You can easily remove the "(Ignored Tests)" registration if you want, and of course adapt the output as you want.

What is very nice with this final version, is that it will take care of Test Classes which have really been loaded/executed, and so you will have a real statistics of what have been executed, instead of a static statistics like you got so far.

Static 'Tests by Category' Computer

If you want, like you asked, to have nothing to do on existing source code, this is a way to perform the Tests by Category computation statically.

This is the StaticTestWithCategoryCounter I wrote for you:

import java.io.File;
import java.lang.annotation.Annotation;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Vector;
import java.util.concurrent.atomic.AtomicInteger;

public class StaticTestWithCategoryCounter {

    public static final String ROOT_DIR_TO_SCAN = "bin";
    public static final String MAIN_TEST_PACKAGES = "misc.category";

    private static final Class<?>[] getClasses(final ClassLoader classLoader)
            throws NoSuchFieldException, SecurityException, IllegalArgumentException, IllegalAccessException {
        Class<?> CL_class = classLoader.getClass();
        while (CL_class != java.lang.ClassLoader.class) {
            CL_class = CL_class.getSuperclass();
        }
        java.lang.reflect.Field ClassLoader_classes_field = CL_class.getDeclaredField("classes");
        ClassLoader_classes_field.setAccessible(true);
        Vector<?> classVector = (Vector<?>) ClassLoader_classes_field.get(classLoader);

        Class<?>[] classes = new Class[classVector.size()]; // Creates an array to avoid concurrent modification
                                                            // exception.
        return classVector.toArray(classes);
    }

    // Registers the information.
    private static final void registerTest(Map<String, AtomicInteger> testByCategoryMap, String category) {
        AtomicInteger count;
        if (testByCategoryMap.containsKey(category)) {
            count = testByCategoryMap.get(category);
        } else {
            count = new AtomicInteger(0);
            testByCategoryMap.put(category, count);
        }

        count.incrementAndGet();
    }


    public static void computeCategoryCounters() throws Exception {
        Map<String, AtomicInteger> testByCategoryMap = new HashMap<>();

        ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        while (classLoader != null) {
            for (Class<?> classToCheck : getClasses(classLoader)) {
                String packageName = classToCheck.getPackage() != null ? classToCheck.getPackage().getName() : "";
                if (!packageName.startsWith(MAIN_TEST_PACKAGES))
                    continue;

                // For each methods of the class.
                for (Method method : classToCheck.getDeclaredMethods()) {
                    Class<?>[] categoryClassToRegister = null;
                    boolean ignored = false;
                    for (Annotation annotation : method.getAnnotations()) {
                        if (annotation instanceof org.junit.experimental.categories.Category) {
                            categoryClassToRegister = ((org.junit.experimental.categories.Category) annotation).value();
                        } else if (annotation instanceof org.junit.Ignore) {
                            ignored = true;

                        } else {
                            // Ignore this annotation.
                            continue;
                        }
                    }

                    if (ignored) {
                        // If you want to compute count of ignored test.
                        registerTest(testByCategoryMap, "(Ignored Tests)");
                    } else if (categoryClassToRegister != null) {
                        for (Class<?> categoryClass : categoryClassToRegister) {
                            registerTest(testByCategoryMap, categoryClass.getCanonicalName());
                        }
                    }

                }

            }
            classLoader = classLoader.getParent();
        }

        System.out.println("\nFinal Statistics:");
        System.out.println("Count of Tests\t\tCategory");
        for (Entry<String, AtomicInteger> info : testByCategoryMap.entrySet()) {
            System.out.println("\t" + info.getValue() + "\t\t" + info.getKey());
        }
    }

    public static List<String> listNameOfAvailableClasses(String rootDirectory, File directory, String packageName) throws ClassNotFoundException {
        List<String> classeNameList = new ArrayList<>();

        if (!directory.exists()) {
            return classeNameList;
        }

        File[] files = directory.listFiles();
        for (File file : files) {           

            if (file.isDirectory()) {
                if (file.getName().contains("."))
                    continue;

                classeNameList.addAll(listNameOfAvailableClasses(rootDirectory, file, packageName));
            } else if (file.getName().endsWith(".class")) {
                String qualifiedName = file.getPath().substring(rootDirectory.length() + 1);
                qualifiedName = qualifiedName.substring(0, qualifiedName.length() - 6).replaceAll(File.separator, ".");

                if (packageName ==null || qualifiedName.startsWith(packageName))
                    classeNameList.add(qualifiedName);
            }
        }

        return classeNameList;
    }

    public static List<Class<?>> loadAllAvailableClasses(String rootDirectory, String packageName) throws ClassNotFoundException {
        List<String> classeNameList = listNameOfAvailableClasses(rootDirectory, new File(rootDirectory), packageName);
        List<Class<?>> classes = new ArrayList<>();

        for (final String className: classeNameList) {
            classes.add(Class.forName(className));
        }

        return classes;
    }

    public static void main(String[] args) {
        try {           
            loadAllAvailableClasses(ROOT_DIR_TO_SCAN, MAIN_TEST_PACKAGES);
            computeCategoryCounters();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

}

You just need to adapt the two constants at beginning to specify:

• where are the (bytecode) classes
• which main package is interesting you (can you set it to null to regard 100% available packages)

The idea of this new version:

• list all classes files matching your 2 constants
• load all corresponding classes
• use untouched source code of dynamic version (now that classes have been loaded)

Let me know if you need further information.