Create my own MethodSorter for junit's @FixMethodOrder

Question

junit 4.11 comes with the @FixMethodOrder-annotation, which makes it possible to select a MethodSorter-implementation to order the junit tests. There are three default sorters, JVM, NAME_ASCENDING and DEFAULT.

Now, I would like to create my own MethodSorter. Can anyone help me with any pointers to how to do that?

Answer 1

Short Answer

This isn't easy, and is discouraged, because JUnit doesn't encourage dependent tests.

Long Answer

For more information, see extended discussion on SortMethodsWith allows the user to choose the order of execution of the methods within a test class.

JUnit doesn't encourage people to write tests which are dependent on other tests. @FixMethodOrder was introduced following a discussion on Sort test methods for predictability.

The basic problem was that Java 7, when using reflection to find methods, doesn't return them in a consistent order. With Java 6, it was pretty much guaranteed that they'd be returned in the order in which they appear in the source file. This is no longer the case.

Some test classes have dependent tests (whether by design or by accident). Sort test methods for predictability at least guarantees that these tests will be executed in a consistent order. However, this ordering is based on the hashCode of the method, so the ordering is deterministic, but hard to predict. So if, you do have a problem with your ordering of tests, then it isn't easy or obvious how to fix the ordering. You have to find a test method name with a higher/lower hashCode.

@FixMethodOrder was introduced to allow the users to fix their ordering problem easily, simply by changing the name. But, for me at least, it's seen as a temporary fix, until the broken tests can be fixed.

Please note also that you can usually specify test execution order from surefire, and there are similar options in ant.

For more information, I've just published a blog post on this very subject.