Pytest ordering of test suites

Question

I've a set of test files (.py files) for different UI tests. I want to run these test files using pytest in a specific order. I used the below command

python -m pytest -vv -s --capture=tee-sys --html=report.html --self-contained-html ./Tests/test_transTypes.py ./Tests/test_agentBank.py ./Tests/test_bankacct.py

The pytest execution is triggered from an AWS Batch job. When the test executions happens it is not executing the test files in the order as specified in the above command. Instead it first runs test_agentBank.py followed by test_bankacct.py, then test_transTypes.py Each of these python files contains bunch of test functions.

I also tried decorating the test function class such as @pytest.mark.run(order=1) in the first python file(test_transTypes.py), @pytest.mark.run(order=2) in the 2nd python file(test_agentBank.py) etc. This seems to run the test in the order, but at the end I get a warning

 PytestUnknownMarkWarning: Unknown pytest.mark.run - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs
.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.run(order=1)

What is the correct way of running tests in a specific order in pytest? Each of my "test_" python files are the ones I need to run using pytest.

Any help much appreciated.

Answer 1

To specify the order in which tests are run in pytest, you can use the pytest-order plugin. This plugin allows you to customize the order in which your tests are run by providing the order marker, which has attributes that define when your tests should run in relation to each other. You can use absolute attributes (e.g., first, second-to-last) or relative attributes (e.g., run this test before this other test) to specify the order of your tests. Here's an example:

import pytest

@pytest.mark.order(2)
def test_foo():
    assert True

@pytest.mark.order(1)
def test_bar():
    assert True

Answer 2

What is the correct way of running tests in a specific order in pytest? Each of my "test_" python files are the ones I need to run using pytest.

Answer to your question is using pytest hooks. You can read about hooks here.

Basically you need to declare a method with specific signature in your conftest.py file. List of hooks that can be defined is stored here.

In your specific case you need a hook named pytest_collection_modifyitems. You can find description of it here. You will be able to change order of elements in items list which is actually list of all collected tests.

So in your conftest.py it will look like this:

def pytest_collection_modifyitems(session, config, items):
   # some code that changing order of elements in items list