pytest: manually add test to discovered tests

Question

# tests/test_assert.py
@pytest.mark.mymark
def custom_assert():
    assert True 

How do I force pytest to discover this test?

In general, how do I dynamically add any test to pytest's list of discovered tests, even if they don't fit in the naming convention?

Answer 1

pytest is fairly customisable, but you'll have to look at its extensive API. Luckily, the code base is statically typed, so you can navigate from functions and classes to other functions and classes fairly easily.

To start off, it pays to understand how pytest discovers tests. Recall the configurable discovery naming conventions:

# content of pytest.ini
# Example 1: have pytest look for "check" instead of "test"
[pytest]
python_files = check_*.py
python_classes = Check
python_functions = *_check

This implies that, for example, the value to python_functions is used somewhere to filter out functions that are not considered as test functions. Do a quick search on the pytest repository to see this:

class PyCollector(PyobjMixin, nodes.Collector):
    def funcnamefilter(self, name: str) -> bool:
        return self._matches_prefix_or_glob_option("python_functions", name)

PyCollector is a base class for pytest Module objects, and module_: pytest.Module has an obj property which is the types.ModuleType object itself. Along with access to the funcnamefilter::name parameter, you can make a subclass of pytest.Module, pytest.Package, and pytest.Class to override funcnamefilter to accept functions decorated your custom @pytest.mark.mymark decorator as test functions:

from __future__ import annotations

import types
import typing as t

import pytest


# Static-type-friendliness
if t.TYPE_CHECKING:
    from _pytest.python import PyCollector

    class _MarkDecorated(t.Protocol):
        pytestmark: list[pytest.Mark]

        def __call__(self, *args: object, **kwargs: object) -> None:
            """Test function callback method"""

else:
    PyCollector: t.TypeAlias = object


def _isPytestMarkDecorated(obj: object) -> t.TypeGuard[_MarkDecorated]:

    """
    Decorating `@pytest.mark.mymark` over a function results in this:

    >>> @pytest.mark.mymark
    ... def f() -> None:
    ...     pass
    ...
    >>> f.pytestmark
    [Mark(name='mymark', args=(), kwargs={})]

    where `Mark` is `pytest.Mark`.

    This function provides a type guard for static typing purposes.
    """

    if (
        callable(obj)
        and hasattr(obj, "pytestmark")
        and isinstance(obj.pytestmark, list)
    ):
        return True
    return False

class _MyMarkMixin(PyCollector):
    def funcnamefilter(self, name: str) -> bool:
        underlying_py_obj: object = self.obj
        assert isinstance(underlying_py_obj, (types.ModuleType, type))
        func: object = getattr(underlying_py_obj, name)
        if _isPytestMarkDecorated(func) and any(
            mark.name == "mymark" for mark in func.pytestmark
        ):
            return True
        return super().funcnamefilter(name)


class MyMarkModule(_MyMarkMixin, pytest.Module):
    pass

The last thing to do is to configure pytest to use your MyMarkModule rather than pytest.Module when collecting test modules. You can do this with the per-directory plugin module file conftest.py, where you would override the hook pytest.pycollect.makemodule (please see pytest's implementation on how to write this properly):

# conftest.py
import typing as t
from <...> import MyMarkModule

if t.TYPE_CHECKING:
    import pathlib
    import pytest


def pytest_pycollect_makemodule(
    module_path: pathlib.Path, parent: object
) -> pytest.Module | None:
    if module_path.name != "__init__.py":
        return MyMarkModule.from_parent(parent, path=module_path)  # type: ignore[no-any-return]

Now you can run pytest <your test file> and you should see all @pytest.mark.mymark functions run as test functions, regardless of whether they're named according to the pytest_functions configuration setting.

---

This is a start on what you need to do, and can do with pytest. You'll have to do this with pytest.Class and pytest.Package as well, if you're planning on using @pytest.mark.mymark elsewhere.