PyTest : Allow Failure Rate

Question

I am currently working on a project where we are running a large suite of parameterized tests (>1M). The tests are randomly generated use-cases and in this large test space, it is expected that in each run certain edge cases will fail, ~1-2%. Is there a implementation for Pytest where you can pass a failure-rate argument, or handle this behavior?

Answer 1

I guess what you want is modify exit status of pytest command, there is a nonpublic hook, named pytest_sessionfinish, could do this.

consider you have following tests:

def test_spam():
    assert 0

def test_ham():
    pass

def test_eggs():
    pass

and a hook in conftest.py:

import pytest, _pytest

ACCEPTABLE_FAILURE_RATE = 50

@pytest.hookimpl()
def pytest_sessionfinish(session, exitstatus):
    if exitstatus != _pytest.main.EXIT_TESTSFAILED:
        return
    failure_rate = (100.0 * session.testsfailed) / session.testscollected
    if failure_rate <= ACCEPTABLE_FAILURE_RATE:
        session.exitstatus = 0

then invoke pytest:

$ pytest --tb=no -q tests.py
F..                                                                                        [100%]
1 failed, 2 passed in 0.06 seconds

here the failure rate is 1 / 3 == 33.3%, below 50%:

$ echo $?
0

you could see the exit status of pytest is 0.