Programmatically collecting or creating tests with py.test

Question

I am writing a pytest plugin to order your tests before running them. I want a better way to write tests for my plugin.

I am using pytest's pytest_collection_modifyitems hook, and then modifying items in-place based on certain markers. The easiest way to test this would be to have an input list of pytest tests, pass them to my function, and assert that the output is in the correct order.

I am having trouble collecting or creating the input test lists. The input tests should be in the same format that pytest will pass in as items to the pytest_collection_modifyitems hook, i.e. they should be instances of _pytest.python.Function with appropriate markers defined.

I will accept answers for either 1) collecting the tests given an existing Python module, or 2) programmatically creating _pytest.python.Function instances that look like what would be passed to the pytest_collection_modifyitems hook. Anything that makes it easy to generate test data.

Answer 1

I'd suggest you to go with higher level testing approach using testdir fixture, which is common for pytest plugins and available via pytester plugin:

pytest_plugins = "pytester"


def test_something(testdir):
    """Test some thing."""
    testdir.makepyfile("""
        import pytest

        @pytest.mark.some
        def test_some():
            assert True

        @pytest.mark.some_thing_else
        def test_some_thing_else():
            assert True
    """)
    result = testdir.runpytest('-vv')
    assert result.stdout.lines[-4:-2] == [
        u'test_something.py::test_some PASSED',
        u'test_something.py::test_some_thing_else PASSED'
    ]

so you can easily have multiple permutations of tests and their markers in the source and by asserting the output lines you assert the actual ordering.