Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using pytest with a src layer

pytest recommends including an additional directory to separate the source code within a project:

my_package
├── src  # <-- no __init__.py on this layer
│   └── my_package
│       ├── __init__.py
│       └── util_module
│           ├── __init__.py
│           └── utils.py
└── tests
    ├── __init__.py
    └── test_util_module
        ├── __init__.py
        └── test_utils.py

Sadly, they say nothing[1] about how imports in the test code should work in such a case, which work for my IDE just fine in this naive example[2], but causes the following error with pytest:

my_package $ pytest

====================== test session starts ======================
platform linux -- Python 3.6.4, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /home/user/workspace/my_package, inifile:
collected 0 items / 1 errors     
                                                                                                                                                                      
============================ ERRORS =============================
___ ERROR collecting tests/test_util_module/test_utils.py ___
ImportError while importing test module '/home/user/workspace/my_package/tests/test_util_module/test_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_util_module/test_utils.py:1: in <module>
    from test.test_module.some_file import starify
E   ModuleNotFoundError: No module named 'my_package.util_module'
!!!! Interrupted: 1 errors during collection !!!!!

I can fix the issue by changing the import of the test to from src.my_package.util_module.utils import starify, but then my IDE complaints about the src part being redundant, so I'd like to keep it out.


[1]: Not the case any more. As of version 3.7.3, pytest recommends the editable install also featured in @hoefling's answer at the top of its good practices.

[2]: Setup is virtualenv env -p python3.6; source env/bin/activate; pip install pytest

like image 927
Arne Avatar asked May 03 '18 12:05

Arne


People also ask

How do I structure a pytest folder?

Project Structure The modules containing pytests should be named “test_*. py” or “*_test.py”. While the pytest discovery mechanism can find tests anywhere, pytests must be placed into separate directories from the product code packages. These directories may either be under the project root or under the Python package.

Is pytest third party?

A number of third-party testing frameworks attempt to address some of the issues with unittest , and pytest has proven to be one of the most popular.

Why pytest is taking too long?

In summary, if you have slow compiling speeds with pytest, try specifying the directory or file your test(s) are in. Or use norecursedirs to skip directories that don't have any tests, like src or . git . in my case it takes 20 seconds to load, even by specifying a test file with 2 unit tests inside.


2 Answers

Recommended approach for pytest>=7: use the pythonpath setting

Recently, pytest has added a new core plugin that supports sys.path modifications via the pythonpath configuration value. The solution is thus much simpler now and doesn't require any workarounds anymore:

pyproject.toml example:

[tool.pytest.ini_options]
pythonpath = [
  "src"
]

pytest.ini example:

[pytest]
pythonpath = src

The path entries are calculated relative to the rootdir, thus the src entry adds path/to/project/src directory to sys.path in this case.

Multiple path entries are also allowed: for a layout

repo/
├── src/
|   └── lib.py
├── src2/
|   └── lib2.py
└── tests
    └── test_lib.py

the configuration

[tool.pytest.ini_options]
pythonpath = [
  "src", "src2",
]

or

[pytest]
pythonpath = src src2

will add both lib and lib2 modules to sys.path, so

import lib
import lib2

will both work.

Original answer

Adjusting the PYTHONPATH (as suggested in the comments) is one possibility to solve the import issue. Another is adding an empty conftest.py file in the src directory:

$ touch src/conftest.py

and pytest will add src to sys.path. This is a simple way to trick pytest into adding codebase to sys.path.

However, the src layout is usually selected when you intend to build a distribution, e.g. providing a setup.py with (in this case) explicitly specifying the root package dir:

from setuptools import find_packages, setup


setup(
    ...
    package_dir={'': 'src'},
    packages=find_packages(where='src'),
    ...
)

and installing the package in the development mode (via python setup.py develop or pip install --editable .) while you're still developing it. This way, your package my_package is correctly integrated in the Python's site packages structure and there's no need to fiddle with PYTHONPATH.

like image 66
hoefling Avatar answered Oct 10 '22 22:10

hoefling


PYTHONPATH updates weren't working for me when using github actions (known prob). Using this pytest-pythonpath install with pytest.ini file worked for me instead:

pip install pytest-pythonpath # accompany with python_path in pytest.ini, so PYTHONPATH is updated with location for modules under test

With this, basic 'pytest' command happily found all tests in subdirs, and found modules under test based on my pytest.ini (set to match source folders in pycharm)

like image 31
Patrick Boundy Avatar answered Oct 10 '22 22:10

Patrick Boundy