How to use pytest fixtures in a decorator without having it as argument on the decorated function

Question

I was trying to use a fixture in a decorator which is intended to decorate test functions. The intention is to provide registered test data to the test. There are two options:

1. Automatic import
2. Manual import

The manual import is trivial. I just need to register the test data globally and can then access it in the test based on its name, the automatic import is trickier, because it should use a pytest fixture.

How shall it look in the end:

@RegisterTestData("some identifier")
def test_automatic_import(): # everything works automatic, so no import fixture is needed
    # Verify that the test data was correctly imported in the test system
    pass

@RegisterTestData("some identifier", direct_import=False)
def test_manual_import(my_import_fixture):
    my_import_fixture.import_all()
    # Verify that the test data was correctly imported in the test system

What did I do:

The decorator registers the testdata globally in a class variable. It also uses the usefixtures marker to mark the test with the respective fixture, in case it doesn't use it. This is necessary because otherwise pytest will not create a my_import_fixture object for the test:

class RegisterTestData:
    # global testdata registry
    testdata_identifier_map = {} # Dict[str, List[str]]

    def __init__(self, testdata_identifier, direct_import = True):
        self.testdata_identifier = testdata_identifier
        self.direct_import = direct_import
        self._always_pass_my_import_fixture = False

    def __call__(self, func):
        if func.__name__ in RegisterTestData.testdata_identifier_map:
            RegisterTestData.testdata_identifier_map[func.__name__].append(self.testdata_identifier)
        else:
            RegisterTestData.testdata_identifier_map[func.__name__] = [self.testdata_identifier]

        # We need to know if we decorate the original function, or if it was already
        # decorated with another RegisterTestData decorator. This is necessary to 
        # determine if the direct_import fixture needs to be passed down or not
        if getattr(func, "_decorated_with_register_testdata", False):
            self._always_pass_my_import_fixture = True
        setattr(func, "_decorated_with_register_testdata", True)

        @functools.wraps(func)
        @pytest.mark.usefixtures("my_import_fixture") # register the fixture to the test in case it doesn't have it as argument
        def wrapper(*args: Any, my_import_fixture, **kwargs: Any):
            # Because of the signature of the wrapper, my_import_fixture is not part
            # of the kwargs which is passed to the decorated function. In case the
            # decorated function has my_import_fixture in the signature we need to pack
            # it back into the **kwargs. This is always and especially true for the
            # wrapper itself even if the decorated function does not have
            # my_import_fixture in its signature
            if self._always_pass_my_import_fixture or any(
                "hana_import" in p.name for p in signature(func).parameters.values()
            ):
                kwargs["hana_import"] = hana_import
            if self.direct_import:
                my_import_fixture.import_all()
            return func(*args, **kwargs)
        return wrapper

This leads to an error in the first test case, as the decorator expects my_import_fixture is passed, but unfortunately it isn't by pytest, because pytest just looks at the signature of the undecorated function.

At this point it becomes hacky since we have to tell pytest to pass my_import_fixture as argument, even though the signature of the original test function does not contain it. We overwrite the pytest_collection_modifyitems hook and manipulate the argnames of the relevant test functions, by adding the fixture name:

def pytest_collection_modifyitems(config: Config, items: List[Item]) -> None:
    for item in items:
        if item.name in RegisterTestData.testdata_identifier_map and "my_import_fixture" not in item._fixtureinfo.argnames:
            # Hack to trick pytest into thinking the my_import_fixture is part of the argument list of the original function
            # Only works because of @pytest.mark.usefixtures("my_import_fixture") in the decorator
            item._fixtureinfo.argnames = item._fixtureinfo.argnames + ("my_import_fixture",)

For completeness a bit code for the import fixture:

class MyImporter:
    def __init__(self, request):
        self._test_name = request.function.__name__
        self._testdata_identifiers = (
            RegisterTestData.testdata_identifier_map[self._test_name]
            if self._test_name in RegisterTestData.testdata_identifier_map
            else []
        )

    def import_all(self):
        for testdata_identifier in self._testdata_identifiers:
            self.import_data(testdata_identifier)

    def import_data(self, testdata_identifier):
        if testdata_identifier not in self._testdata_identifiers: #if someone wants to manually import single testdata
            raise Exception(f"{import_metadata.identifier} is not registered. Please register it with the @RegisterTestData decorator on {self._test_name}")
        # Do the actual import logic here


@pytest.fixture
def my_import_fixture(request /*some other fixtures*/):
    # Do some configuration with help of the other fixtures
    importer = MyImporter(request)
    try:
        yield importer
    finally:
        # Do some cleanup logic

Now my question is if there is a better (more pytest native) way to do this. There was a similar question asked before, but it never got answered, I will link my question to it, because it essentially describes a hacky way how to solve it (at least with pytest 6.1.2 and python 3.7.1 behaviour).

Some might argue, that I could remove the fixture and create a MyImporter object in the decorator. I would then face the same issue with the request fixture, but could simply avoid this by passing func.__name__ instead of the request fixture to the constructor.

Unfortunately, this falls apart because of the configuration and cleanup logic I have in my_import_fixture. Of course I could replicate that, but it becomes super complex, because I use other fixtures which also have some configuration and cleanup logic and so on. Also in the end this would be duplicated code which needs to be kept in sync.

I also don't want my_import_fixture to be autouse because it implies some requirements for the test.

Answer 1

I hope this answer is helpful to someone over a year later. The underlying problem is that when you do

  @functools.wraps(func)
  def wrapper(*args: Any, my_import_fixture, **kwargs: Any):
    . . .

The signature of wrapper is whatever the signature of func was. my_import_fixture isn't part of the signature. Once I understood this was the problem, I got a very helpful answer about how to fix it quite quickly here How can I wrap a python function in a way that works with with inspect.signature?

To get pytest to pass my_import_fixture to your wrapper, do something like this:

  @functools.wraps(func)
  def wrapper(*args: Any, **kwargs: Any):
    # Pytest will pass `my_import_fixture` in kwargs.
    # If the wrapped func needs `my_import_fixture` then we need to keep it in kwargs.
    # If func doesn't expect a `my_import_fixture` argument then we need to remove it from kwargs.
    if 'my_import_fixture' in inspect.signature(func).parameters.keys():
      my_import_fixture = kwargs['my_import_fixture']
    else:
      my_import_fixture = kwargs.pop('my_import_fixture')

    # Do whatever it is you need to do with `my_import_fixture` here.
    # I'm omitting that specific logic from this answer
    # . . .

    # Now call the wrapped func with the correct arguments
    return func(*args, **kwargs)

  # If the wrapped func already uses the `my_import_fixture` fixture
  # then we don't need to do anything.
  # If it doesn't use `my_import_fixture` we need to add it to the signature
  # in a way that pytest will notice.
  if 'my_import_fixture' not in inspect.signature(func).parameters.keys():
    original_signature = inspect.signature(func)
    wrapper.__signature__ = original_signature.replace(
      parameters=(
        list(original_signature.parameters.values()) +
        [inspect.Parameter('my_import_fixture', inspect.Parameter.POSITIONAL_OR_KEYWORD)]
      )
    )

  return wrapper

PEP-362 explains how this works. Credit to @Andrej Kesely who answered the linked question.

Apologies - I've simplified the code a little bit because the problem I solved was slightly different from your problem (I needed the wrapper to access the request fixture even if the wrapped test case didn't use it). The same solution should work for you though.