How to run the same TestSuite multiple times unittest texttestrunner

Question

I want to run tests with multiple builds of a product running them once. Here is example of the code:

import unittest

suite = unittest.TestLoader().discover("./tests")
runner = unittest.TextTestRunner()

for build in [build1, build2]:
    get_the_build(build)
    runner.run(suite)

The first iteration works well, but on the start of the second one an error appears:

Traceback (most recent call last):
  File "D:/Path/to/my/folder/run_tests.py", line 9, in <module>
    runner.run(suite)
  File "C:\Program Files (x86)\Python36-32\lib\unittest\runner.py", line 176, in run
    test(result)
  File "C:\Program Files (x86)\Python36-32\lib\unittest\suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "C:\Program Files (x86)\Python36-32\lib\unittest\suite.py", line 122, in run
    test(result)
TypeError: 'NoneType' object is not callable

What is happening? What result runner calls? And why does it fail? Any ideas how to solve the problem?

Answer 1

Well, well, well. I have spent the last hour of my life looking at the code of unittest in GitHub, which can be found here. I just went to the code of suite.py (here), one of the files in the error you are getting. This is the actual code of TestSuite.run:

def run(self, result, debug=False):
    topLevel = False
    if getattr(result, '_testRunEntered', False) is False:
        result._testRunEntered = topLevel = True

    for index, test in enumerate(self):
        if result.shouldStop:
            break

        if _isnotsuite(test):
            self._tearDownPreviousClass(test, result)
            self._handleModuleFixture(test, result)
            self._handleClassSetUp(test, result)
            result._previousTestClass = test.__class__

            if (getattr(test.__class__, '_classSetupFailed', False) or
                getattr(result, '_moduleSetUpFailed', False)):
                continue

        if not debug:
            test(result)
        else:
            test.debug()

        if self._cleanup:
            self._removeTestAtIndex(index)

    if topLevel:
        self._tearDownPreviousClass(None, result)
        self._handleModuleTearDown(result)
        result._testRunEntered = False
    return result

So, basically, what this code does is to iterate over each test the suite has and invoke it:

for index, test in enumerate(self):
    ...
    if not debug:
        test(result)  # This is the line throwing the error
    ...

As you can see, that loop iterates over the suite itself, so I it must have an __iter__ method defined somewhere. After 5 minutes of not finding it inside the class TestSuite, I realized is the parent class who has such method. This is what I found in BaseTestSuite:

def __iter__(self):
    return iter(self._tests)

Basically, it just returns an iterator of the tests. In that moment, such line of code, was a high wall I couldn't surepass. But I didn't give up and went back to TestSuite.run definition and, miraculously, I spotted the next lines:

...
if self._cleanup:
    self._removeTestAtIndex(index)
...

And that made me wonder: "Are the tests being removed? Let me investigate". Then I was enlightened, because inside _removeTestAtIndex I spotted this line:

self._tests[index] = None

End of story. So, after running all of your tests the first time, they got converted into nothing more than None: the list of tests inside the suite ended up being a list of Nones ([None, None, ..., None]).

So, how do you prevent such behaviour? Just turn off the _cleanup flag inside the suite. This should work:

Answer

import unittest

suite = unittest.TestLoader().discover("./tests")
suite._cleanup = False  # Prevent such cleanup

runner = unittest.TextTestRunner()

for build in [build1, build2]:
    get_the_build(build)
    runner.run(suite)

Sorry for the long story but, besides showing you how to solve your issue, I also wanted to teach you how to debug whatever.

Let me know if this actually worked for you. Otherwise, tell me what went wrong.

Answer 2

Feels like a horrible hack, but making a deep copy of the suite each time before running it solved the problem for me:

import unittest
import copy

suite = unittest.TestLoader().discover("./tests")
runner = unittest.TextTestRunner()

for build in [build1, build2]:
    get_the_build(build)
    runner.run(copy.deepcopy(suite))