How to test the pytest fixture itself?

Question

What is the proper way to test that the pytest fixture itself. Please do not confuse it with using fixture in tests. I want just tests the fixtures correctness by itself.

When trying to call and execute them inside test I am facing:

Fixture "app" called directly. Fixtures are not meant to be called directly

Any input on that will be appreciated. Docs on this topic are not giving me meaningful guidance: https://docs.pytest.org/en/latest/deprecations.html#calling-fixtures-directly

The motivation for testing fixtures itself come to me, because when our test were failing due to bug in fixture this wasn't tracked correctly in our TAP files, what motivated me to test the fixtures stand alone.

Answer 1

pytest has a pytester plugin that was made for the purpose of testing pytest itself and plugins; it executes tests in an isolated run that doesn't affect the current test run. Example:

# conftest.py

import pytest

pytest_plugins = ['pytester']

@pytest.fixture
def spam(request):
    yield request.param

The fixture spam has an issue that it will only work with parametrized tests; once it is requested in an unparametrized test, it will raise an AttributeError. This means that we can't test it via a regular test like this:

def test_spam_no_params(spam):
    # too late to verify anything - spam already raised in test setup!
    # In fact, the body of this test won't be executed at all.
    pass

Instead, we execute the test in an isolated test run using the testdir fixture which is provided by the pytester plugin:

import pathlib
import pytest


# an example on how to load the code from the actual test suite
@pytest.fixture
def read_conftest(request):
    return pathlib.Path(request.config.rootdir, 'conftest.py').read_text()


def test_spam_fixture(testdir, read_conftest):
    # you can create a test suite by providing file contents in different ways, e.g.
    testdir.makeconftest(read_conftest)
    testdir.makepyfile(
        """
        import pytest

        @pytest.mark.parametrize('spam', ('eggs', 'bacon'), indirect=True)
        def test_spam_parametrized(spam):
            assert spam in ['eggs', 'bacon']

        def test_spam_no_params(spam):
            assert True
""")
    result = testdir.runpytest()
    # we should have two passed tests and one failed (unarametrized one)
    result.assert_outcomes(passed=3, error=1)
    # if we have to, we can analyze the output made by pytest
    assert "AttributeError: 'SubRequest' object has no attribute 'param'" in ' '.join(result.outlines)

Another handy possibility of loading test code for the tests is the testdir.copy_example method. Setup the root path in the pytest.ini, for example:

[pytest]
pytester_example_dir = samples_for_fixture_tests
norecursedirs = samples_for_fixture_tests

Now create the file samples_for_fixture_tests/test_spam_fixture/test_x.py with the contents:

import pytest

@pytest.mark.parametrize('spam', ('eggs', 'bacon'), indirect=True)
def test_spam_parametrized(spam):
    assert spam in ['eggs', 'bacon']

def test_spam_no_params(spam):
    assert True

(it's the same code that was passed as string to testdir.makepyfile before). The above test changes to:

def test_spam_fixture(testdir, read_conftest):
    testdir.makeconftest(read_conftest)
    # pytest will now copy everything from samples_for_fixture_tests/test_spam_fixture
    testdir.copy_example()
    testdir.runpytest().assert_outcomes(passed=3, error=1)

This way, you don't have to maintain Python code as string in tests and can also reuse existing test modules by running them with pytester. You can also configure test data roots via the pytester_example_path mark:

@pytest.mark.pytester_example_path('fizz')
def test_fizz(testdir):
    testdir.copy_example('buzz.txt')

will look for the file fizz/buzz.txt relative to the project root dir.

For more examples, definitely check out the section Testing plugins in pytest docs; also, you may find my other answer to the question How can I test if a pytest fixture raises an exception? helpful as it contains yet another working example to the topic. I have also found it very helpful to study the Testdir code directly as sadly pytest doesn't provide an extensive docs for it, but the code is pretty much self-documenting.