Pytest: running tests multiple times with different input data

Question

I want to run through a collection of test functions with different fixtures for each run. Generally, the solutions suggested on Stack Overflow, documentation and in blog posts fall under two categories. One is by parametrizing the fixture:

@pytest.fixture(params=list_of_cases)
def some_case(request):
    return request.param

The other is by calling metafunc.parametrize in order to generate multiple tests:

def pytest_generate_tests(metafunc):
    metafunc.parametrize('some_case', list_of_cases)

The problem with both approaches is the order in which the cases are run. Basically it runs each test function with each parameter, instead of going through all test functions for a given parameter and then continuing with the next parameter. This is a problem when some of my fixtures are comparatively expensive database calls.

To illustrate this, assume that dataframe_x is another fixture that belongs to case_x. Pytest does this

test_01(dataframe_1)
test_01(dataframe_2)
...
test_50(dataframe_1)
test_50(dataframe_2)

instead of

test_01(dataframe_1)
...
test_50(dataframe_1)

test_01(dataframe_2)
...
test_50(dataframe_2)

The result is that I will fetch each dataset from the DB 50 times instead of just once. Since I can only define the fixture scope as 'session', 'module' or 'function', I couldn't figure out how to group my tests to that they are run together in chunks.

Is there a way to structure my tests so that I can run through all my test functions in sequence for each dataset?

Answer 1

If you only want to load the dataframes once you could use the scope parameter with 'module' or 'session'.

@pytest.fixture(scope="module", params=[1, 2])
def dataframe(request):
    if request.param == 1:
        return #load datagrame_1
    if request.param == 2:
        return #load datagrame_2

The tests will still be run alternately but the dataframe will only be loaded once per module or session.