Isolating pytest tests from each other

Question

I'm working on a rapidly growing Python project. Lately, our test suite started being somewhat unmanageable. Some tests fail when the modules they are in get executed in the wrong order, despite being seemingly well-isolated.

I found some other questions about this, but they were concerned with fixtures:

Pytest fixtures interfering with each other

test isolation between pytest-hypothesis runs

While we're using fixtures as well, I do not think the problem lies within them, but more likely in the fact that we use libraries where classes have internal state which gets changed by the test run, for example by mockito-python.

I originally come from Java world where this does not happen unless you explicitly make your tests depend on each other or do some crazy and unusual things. Following the same set of practices in Python led me to these problems, so I realize I might be missing some crucial rule of Python test development.

Is it possible to tell pytest to drop/revert/reinitialize the internal state of all classes between various module runs?

If not, which rules should we follow during test development to prevent these issues?

Edit: One of the issues we identified was that we had some objects set up at top level in the test file, which were created by mockito-python. When the tests were executed, the files are first all imported and then the tests are executed. What happened was that one test was calling mockito.unstub() which tears down all mocks in mockito's mock registry. So when the test was actually executed, another test had already torn down its mocks. This behavior is quite counter-intuitive, especially for unexperienced developers.

Answer 1

In python it can happen easily that some state is modified by mistake. For example, when assigning a list to a variable, it is easy enough to forget to add a [:] when it is desired that a copy of the list is created.

x = [0,1,2,3,4,5]
y = x    # oops, should have been x[:]
y[2] = 7 # now we modify state somewhere...
x
=> [0, 1, 7, 3, 4, 5]

One possible approach to at least more likely identify such problems could be to execute your unit-tests in random order. I ran an experiment building upon the idea from https://stackoverflow.com/a/4006044/5747415:

import unittest
import random
def randcmp(_, x, y):
    return random.randrange(-1, 2)
unittest.TestLoader.sortTestMethodsUsing = randcmp

As a result, the execution order of tests changed between the test executions. If by mistake your tests happen to have dependencies, you might be able to figure it out this way, because certain execution orders would lead to failures. Certainly, you would start on a small scale (only executing a small amount of tests), so you have the chance to more easily find the culprit.

Maybe worth trying...

Answer 2

Is it possible to tell pytest to drop/revert/reinitialize the internal state of all classes between various module runs?

You can achieve this by using importlib to both invalidate cached modules and reload imported modules (thus refreshing the code & state).

This can be useful when code is required to run at the module level, for example in Flask where it's common to initialize app at the module level in main.py:

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "<p>Hello, World!</p>"

In this case app will be created whenever you import from main import .... You may want to test initializing app in different ways, so in your test you could do something like

def test_app():
    import main
    importlib.reload(main)  # reset module state
    # do something with main.app

This would reset the state of the main module during your test.