JUnit test class order

Question

I have a java app with maven. Junit for tests, with failsafe and surefire plugins. I have more than 2000 integration tests. To speed up the test running, I use failsafe jvmfork to run my tests parallel. I have some heavy test class, and they typically running at end of my test execution and it is slows down my CI verify process. The filesafe runorder:balanced would be a good option for me, but i cant use it because the jvmfork. To rename the test classes or move to another package and run it alpahabetical is not an option. Any suggestion how can I run my slow test classes at the begining of the verify process?

Answer 1

In JUnit 5 (from version 5.8.0 onwards) test classes can be ordered too.

src/test/resources/junit-platform.properties:

# ClassOrderer$OrderAnnotation sorts classes based on their @Order annotation
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$OrderAnnotation

Other Junit built-in class orderer implementations:

org.junit.jupiter.api.ClassOrderer$ClassName
org.junit.jupiter.api.ClassOrderer$DisplayName
org.junit.jupiter.api.ClassOrderer$Random

^{For other ways (beside junit-platform.properties file) to set configuration parameters see JUnit 5 user guide.}

You can also provide your own orderer. It must implement ClassOrderer interface:

package foo;
public class MyOrderer implements ClassOrderer {
    @Override
    public void orderClasses(ClassOrdererContext context) {
        Collections.shuffle(context.getClassDescriptors());
    }
}

junit.jupiter.testclass.order.default=foo.MyOrderer

Note that @Nested test classes cannot be ordered by a ClassOrderer.

Refer to JUnit 5 documentations and ClassOrderer API docs to learn more about this.

Answer 2

I gave the combination of answers I found a try:

• Running JUnit4 Test classes in specified order
  
• Running JUnit Test in parallel on Suite Level

The second answer is based on these classes of this github project, which is available under the BSD-2 license.

I defined a few test classes:

public class LongRunningTest {

    @Test
    public void test() {

        System.out.println(Thread.currentThread().getName() + ":\tlong test - started");

        long time = System.currentTimeMillis();
        do {
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
            }
        } while(System.currentTimeMillis() - time < 1000);

        System.out.println(Thread.currentThread().getName() + ":\tlong test - done");
    }
}

@Concurrent
public class FastRunningTest1 {

    @Test
    public void test1() {
        try {
            Thread.sleep(250);
        } catch (InterruptedException e) {
        }

        System.out.println(Thread.currentThread().getName() + ":\tfrt1-test1 - done");
    }

    // +7 more repetions of the same method
}

Then I defined the test suites:
(FastRunningTest2 is a copy of the first class with adjusted output)

@SuiteClasses({LongRunningTest.class, LongRunningTest.class})
@RunWith(Suite.class)
public class SuiteOne {}

@SuiteClasses({FastRunningTest1.class, FastRunningTest2.class})
@RunWith(Suite.class)
public class SuiteTwo {}

@SuiteClasses({SuiteOne.class, SuiteTwo.class})
@RunWith(ConcurrentSuite.class)
public class TopLevelSuite {}

When I execute the TopLevelSuite I get the following output:

TopLevelSuite-1-thread-1: long test - started FastRunningTest1-1-thread-4: frt1-test4 - done FastRunningTest1-1-thread-2: frt1-test2 - done FastRunningTest1-1-thread-1: frt1-test1 - done FastRunningTest1-1-thread-3: frt1-test3 - done FastRunningTest1-1-thread-5: frt1-test5 - done FastRunningTest1-1-thread-3: frt1-test6 - done FastRunningTest1-1-thread-1: frt1-test8 - done FastRunningTest1-1-thread-5: frt1-test7 - done FastRunningTest2-2-thread-1: frt2-test1 - done FastRunningTest2-2-thread-2: frt2-test2 - done FastRunningTest2-2-thread-5: frt2-test5 - done FastRunningTest2-2-thread-3: frt2-test3 - done FastRunningTest2-2-thread-4: frt2-test4 - done TopLevelSuite-1-thread-1: long test - done TopLevelSuite-1-thread-1: long test - started FastRunningTest2-2-thread-5: frt2-test8 - done FastRunningTest2-2-thread-2: frt2-test6 - done FastRunningTest2-2-thread-1: frt2-test7 - done TopLevelSuite-1-thread-1: long test - done

Which basically shows that the LongRunningTest is executed in parralel to the FastRunningTests. The default value of threads used for parallel execution defined by the Concurrent Annotation is 5, which can be seen in the output of the parallel execution of the FastRunningTests.

The downside is that theses Threads are not shared between FastRunningTest1 and FastRunningTest2.

---

This behavious shows that it is "somewhat" possible to do what you want to do (so whether that works with your current setup is a different question).

Also I am not sure whether this is actually worth the effort,

• as you need to prepare those TestSuites manually (or write something that autogenerates them)
• and you need to define the Concurrent Annotation for all those classes (maybe with a different number of threads for each class)

---

As this basically shows that it is possible to define the execution order of classes and trigger their parallel execution, it should also be possibly to get the whole process to only use one ThreadPool (but I am not sure what the implication of that would be).

As the whole concept is based on a ThreadPoolExecutor, using a PriorityBlockingQueue which gives long running tasks a higher priority you would get closer to your ideal outcome of executing the long running tests first.

---

I experimented around a bit more and implemented my own custom suite runner and junit runner. The idea behind is to have your JUnitRunner submit the tests into a queue which is handeld by a single ThreadPoolExecutor. Because I didn't implement a blocking operation in the RunnerScheduler#finish method, I ended up with a solution where the tests from all classes were passed to the queue before the execution even started. (That might look different if there a more test classes and methods involved).

At least it proves the point that you can mess with junit at this level if you really want to.

The code of my poc is a bit messy and to lengthy to put it here, but if someone is interested I can push it into a github project.

Answer 3

In out project we had created a few marker interfaces ( example

public interface SlowTestsCategory {}

)

and put it into the @Category annotation of JUnit in the test class with slow tests.

@Category(SlowTestsCategory.class)

After that we created some special tasks for Gradle to run tests by category or a few categories by custom order:

task unitTest(type: Test) {
  description = 'description.'
  group = 'groupName'

  useJUnit {
    includeCategories 'package.SlowTestsCategory'
    excludeCategories 'package.ExcludedCategory'
  }
}

This solution is served by Gradle, but maybe it'll be helpful for you.