How to run unittest test cases in the order they are declared

Question

I fully realize that the order of unit tests should not matter. But these unit tests are as much for instructional use as for actual unit testing, so I would like the test output to match up with the test case source code.

I see that there is a way to set the sort order by setting the sortTestMethodsUsing attribute on the test loader. The default is a simple cmp() call to lexically compare names. So I tried writing a cmp-like function that would take two names, find their declaration line numbers and them return the cmp()-equivalent of them:

import unittest

class TestCaseB(unittest.TestCase):
    def test(self):
        print("running test case B")

class TestCaseA(unittest.TestCase):
    def test(self):
        print("running test case A")

import inspect
def get_decl_line_no(cls_name):
    cls = globals()[cls_name]
    return inspect.getsourcelines(cls)[1]

def sgn(x):
    return -1 if x < 0 else 1 if x > 0 else 0

def cmp_class_names_by_decl_order(cls_a, cls_b):
    a = get_decl_line_no(cls_a)
    b = get_decl_line_no(cls_b)
    return sgn(a - b)

unittest.defaultTestLoader.sortTestMethodsUsing = cmp_class_names_by_decl_order
unittest.main()

When I run this, I get this output:

running test case A
.running test case B
.
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

indicating that the test cases are not running in the declaration order.

My sort function is just not being called, so I suspect that main() is building a new test loader, which is wiping out my sort function.

Answer 1

The solution is to create a TestSuite explicitly, instead of letting unittest.main() follow all its default test discovery and ordering behavior. Here's how I got it to work:

import unittest

class TestCaseB(unittest.TestCase):
    def runTest(self):
        print("running test case B")

class TestCaseA(unittest.TestCase):
    def runTest(self):
        print("running test case A")


import inspect
def get_decl_line_no(cls):
    return inspect.getsourcelines(cls)[1]

# get all test cases defined in this module
test_case_classes = list(filter(lambda c: c.__name__ in globals(), 
                                unittest.TestCase.__subclasses__()))

# sort them by decl line no
test_case_classes.sort(key=get_decl_line_no)

# make into a suite and run it
suite = unittest.TestSuite(cls() for cls in test_case_classes)
unittest.TextTestRunner().run(suite)

This gives the desired output:

running test case B
.running test case A
.
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

It is important to note that the test method in each class must be named runTest.

Answer 2

You can manually build a TestSuite where your TestCases and all tests inside them run by line number:

# Python 3.8.3
import unittest
import sys
import inspect


def isTestClass(x):
    return inspect.isclass(x) and issubclass(x, unittest.TestCase)


def isTestFunction(x):
    return inspect.isfunction(x) and x.__name__.startswith("test")


class TestB(unittest.TestCase):
    def test_B(self):
        print("Running test_B")
        self.assertEqual((2+2), 4)

    def test_A(self):
        print("Running test_A")
        self.assertEqual((2+2), 4)

    def setUpClass():
        print("TestB Class Setup")


class TestA(unittest.TestCase):
    def test_A(self):
        print("Running test_A")
        self.assertEqual((2+2), 4)

    def test_B(self):
        print("Running test_B")
        self.assertEqual((2+2), 4)

    def setUpClass():
        print("TestA Class Setup")


def suite():

    # get current module object
    module = sys.modules[__name__]

    # get all test className,class tuples in current module
    testClasses = [
        tup for tup in
        inspect.getmembers(module, isTestClass)
    ]

    # sort classes by line number
    testClasses.sort(key=lambda t: inspect.getsourcelines(t[1])[1])

    testSuite = unittest.TestSuite()

    for testClass in testClasses:
        # get list of testFunctionName,testFunction tuples in current class
        classTests = [
            tup for tup in
            inspect.getmembers(testClass[1], isTestFunction)
        ]

        # sort TestFunctions by line number
        classTests.sort(key=lambda t: inspect.getsourcelines(t[1])[1])

        # create TestCase instances and add to testSuite;
        for test in classTests:
            testSuite.addTest(testClass[1](test[0]))

    return testSuite


if __name__ == '__main__':

    runner = unittest.TextTestRunner()
    runner.run(suite())

Output:

TestB Class Setup
Running test_B
.Running test_A
.TestA Class Setup
Running test_A
.Running test_B
.
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

Answer 3

As stated in the name, sortTestMethodsUsing is used to sort test methods. It is not used to sort classes. (It is not used to sort methods in different classes either; separate classes are handled separately.)

If you had two test methods in the same class, sortTestMethodsUsing would be used to determine their order. (At that point, you would get an exception because your function expects class names.)