Run only tests which depend on the change

Question

How to detect which tests pytest need to run?

Up to now I run all tests in CI. This is slow.

Isn't there a way to automatically detect which tests need to run, and only execute a fraction of the whole test suite?

Background: I am reading the Google SE book and read about their gigantic monorepo and that their tool Blaze can detect which tests need to run. Sounds very cool.

Answer 1

I had good experience with pytest-testmon which determines the subset of tests to execute on code change based on test coverage. The details on that are explained well in Determining affected tests section of testmon's docs, but the general advice is to maintain high line/branch coverage for testmon to be effective. This also means that it will fail where coverage fails.

If you want to use it in the CI, you need to cache the .testmondata database file between test jobs and ensure the --testmon arg is used when invoking pytest. It also depends on your CI provider, but most popular ones out there offer caching mechanisms (Github, Gitlab, CircleCI, Appveyor etc).

testmon can also be effectively combined with pytest-watch watchdog to keep a daemon for test execution running. An example command that reruns subset of tests on code changes and sends system notifications on success or failure:

$ pytest-watch --runner="pytest --testmon" \
  --onfail="notify-send --urgency=low -i error 'Tests Failed'" \
  --onpass="notify-send --urgency=low -i info 'Tests passed'"

You can also use Bazel for building and testing as well; however, IMO it is an overkill for a pure Python project and is not worth it for just the incremental testing purposes.

Answer 2

You can use the pytest-incremental plugin for this which uses AST to analyse the import structure of a project and only runs tests for which the source code has changed.

pip install pytest-incremental
python -m pytest --inc

Analyzing the graph is easy to see that a change in app would cause only test_app to be execute. And a change in util would cause all tests to be executed.

from the docs

Answer 3

You can add @pytest.mark to the tests

@pytest.mark.SomeFeature
@pytest.mark.Regression
def test_example1(self):
    pass

@pytest.mark.AnotherFeature
@pytest.mark.Regression
def test_example2(self):
    pass

And use it when triggering the tests

pytest TestsFolder -m SomeFeature

This will run only test_example1, -m Regression will run both tests.