Layout and importing for pytest in python3

Question

I'm having trouble importing modules from my pytest functions. I know there's a million questions on this, but I've read through a bunch, and I'm still having trouble understanding.

$ tree
.
└── code
    ├── eight_puzzle.py
    ├── missionaries_and_cannibals.py
    ├── node.py
    ├── search.py
    └── test
        ├── test_eight_puzzle.py
        └── test_search.py

2 directories, 6 files
$
$ grep import code/test/test_search.py
import sys
import pytest
import code.search
$
$ pytest
...
ImportError while importing test module '~/Documents/code/test/test_search.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
code/test/test_search.py:14: in <module>
    import code.search
E   ModuleNotFoundError: No module named 'code.search'; 'code' is not a package
...

I expected that to work. 'code' is a package, right? A package in Python 3 is any directory with .py files in it.

I've also tried it with a relative import - from .. import search - and I get the following.

ImportError while importing test module '~/Documents/code/test/test_search.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
code/test/test_search.py:14: in <module>
    from .. import search
E   ImportError: attempted relative import with no known parent package

I've also tried modifying sys.path as shown here, specifying my PYTHONPATH, and adding init.py files in code and test.

Can I get this import to work without using something like setuptools? This is just for experimenting, so I'd rather not deal with the overhead.

It may also be important to note that I'm using conda, because it seems to work when I'm using the python 2 pip-installed version of pytest with init.py files.

Answer 1

Some notes about directories without __init__.py files first:

Implicit namespace packages

Although a directory without an __init__.py is a valid import source in Python 3, it is not a regular package, rather being an implicit namespace package (see PEP 420 for the details). Among other properties, implicit namespace packages are second-class citizens when it comes to importing, meaning that when Python has two packages with the same name in sys.path, one being a regular package and another being an implicit namespace package, the regular one will be preferred regardless what package comes first. Check it yourself:

$ mkdir -p implicit_namespace/mypkg
$ echo -e "def spam():\n    print('spam from implicit namespace package')" > implicit_namespace/mypkg/mymod.py
$ mkdir -p regular/mypkg
$ touch regular/mypkg/__init__.py
$ echo -e "def spam():\n    print('spam from regular package')" > regular/mypkg/mymod.py
$ PYTHONPATH=implicit_namespace:regular python3 -c "from mypkg.mymod import spam; spam()"

This will print spam from regular package: although implicit_namespace comes first in sys.path, mypkg.mymod from regular is imported instead because regular/mypkg is a regular package.

---

Now you know that since your package code is an implicit namespace package, Python will prefer regular imports of code to yours if it encounters one. Unfortunately for you, there is a module code in the stdlib, so it's practically a "reverse name shadowing" problem: you have an import object with the same name as the one from stdlib, but instead of shadowing the stdlib import, it shadows yours.

You thus need to do two things in order to make your layout usable:

1. give the code dir a unique name (let it be mycode for this answer's example)
2. after that, you still need to fix the sys.path when running the tests from the project root dir because it's not in sys.path per se. You have some possibilities:
   • add an empty conftest.py file to the root dir (aside the mycode dir). This will instruct pytest to add the root dir to sys.path (see here for an explanation). You can now just run pytest as usual and the imports will be resolved;
   • run the tests via python -m pytest - invoking interpreter directly adds the current dir to sys.path;
   • add the current dir to sys.path via PYTHONPATH env var, e.g. run PYTHONPATH=. pytest.