How do I concisely implement multiple similar unit tests in the Python unittest framework?

Question

I'm implementing unit tests for a family of functions that all share a number of invariants. For example, calling the function with two matrices produce a matrix of known shape.

I would like to write unit tests to test the entire family of functions for this property, without having to write an individual test case for each function (particularly since more functions might be added later).

One way to do this would be to iterate over a list of these functions:

import unittest
import numpy

from somewhere import the_functions
from somewhere.else import TheClass

class Test_the_functions(unittest.TestCase):
  def setUp(self):
    self.matrix1 = numpy.ones((5,10))
    self.matrix2 = numpy.identity(5)

  def testOutputShape(unittest.TestCase):
     """Output of functions be of a certain shape"""
     for function in all_functions:
       output = function(self.matrix1, self.matrix2)
       fail_message = "%s produces output of the wrong shape" % str(function)
       self.assertEqual(self.matrix1.shape, output.shape, fail_message)

if __name__ == "__main__":
  unittest.main()

I got the idea for this from Dive Into Python. There, it's not a list of functions being tested but a list of known input-output pairs. The problem with this approach is that if any element of the list fails the test, the later elements don't get tested.

I looked at subclassing unittest.TestCase and somehow providing the specific function to test as an argument, but as far as I can tell that prevents us from using unittest.main() because there would be no way to pass the argument to the testcase.

I also looked at dynamically attaching "testSomething" functions to the testcase, by using setattr with a lamdba, but the testcase did not recognize them.

How can I rewrite this so it remains trivial to expand the list of tests, while still ensuring every test is run?

Answer 1

Here's my favorite approach to the "family of related tests". I like explicit subclasses of a TestCase that expresses the common features.

class MyTestF1( unittest.TestCase ):
    theFunction= staticmethod( f1 )
    def setUp(self):
        self.matrix1 = numpy.ones((5,10))
        self.matrix2 = numpy.identity(5)
    def testOutputShape( self ):
        """Output of functions be of a certain shape"""
        output = self.theFunction(self.matrix1, self.matrix2)
        fail_message = "%s produces output of the wrong shape" % (self.theFunction.__name__,)
        self.assertEqual(self.matrix1.shape, output.shape, fail_message)

class TestF2( MyTestF1 ):
    """Includes ALL of TestF1 tests, plus a new test."""
    theFunction= staticmethod( f2 )
    def testUniqueFeature( self ):
         # blah blah blah
         pass

class TestF3( MyTestF1 ):
    """Includes ALL of TestF1 tests with no additional code."""
    theFunction= staticmethod( f3 )

Add a function, add a subclass of MyTestF1. Each subclass of MyTestF1 includes all of the tests in MyTestF1 with no duplicated code of any kind.

Unique features are handled in an obvious way. New methods are added to the subclass.

It's completely compatible with unittest.main()