Count subtests in Python unittests separately

Question

Since version 3.4, Python supports a simple subtest syntax when writing unittests. A simple example could look like this:

import unittest

class NumbersTest(unittest.TestCase):

    def test_successful(self):
        """A test with subtests that will all succeed."""
        for i in range(0, 6):
            with self.subTest(i=i):
                self.assertEqual(i, i)

if __name__ == '__main__':
    unittest.main()

When running the tests, the output will be

python3 test_foo.py --verbose
test_successful (__main__.NumbersTest)
A test with subtests that will all succeed. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

However, in my real world use cases, the subtests will depend on a more complex iterable and check something which is very different for each subtest. Consequently, I would rather have each subtest counted and listed as a separated test case in the output (Ran 6 tests in ... in this example) to get the full picture.

Is this somehow possible with the plain unittest module in Python? The nose test generator feature would output each test separately but I would like to stay compatible with the standard library if possible.

Answer 1

You could subclass unittest.TestResult:

class NumbersTestResult(unittest.TestResult):
    def addSubTest(self, test, subtest, outcome):
        # handle failures calling base class
        super(NumbersTestResult, self).addSubTest(test, subtest, outcome)
        # add to total number of tests run
        self.testsRun += 1

Then in NumbersTest override the run function:

def run(self, test_result=None):
    return super(NumbersTest, self).run(NumbersTestResult())

Sorry I cannot test this in a fully working environment right now, but this should do the trick.

Answer 2

Using python 3.5.2, themiurge's answer didn't work out-of-the-box for me but a little tweaking got it to do what I wanted.

I had to specifically get the test runner to use this new class as follows:

if __name__ == '__main__': 
    unittest.main(testRunner=unittest.TextTestRunner(resultclass=NumbersTestResult))

However this didn't print the details of the test failures to the console as in the default case. To restore this behaviour I had to change the class NumbersTestResult inherited from to unittest.TextTestResult.

class NumbersTestResult(unittest.TextTestResult):
    def addSubTest(self, test, subtest, outcome):
        # handle failures calling base class
        super(NumbersTestResult, self).addSubTest(test, subtest, outcome)
        # add to total number of tests run
        self.testsRun += 1