Why is relative path not working in python tests?

Question

My directory layout is as follows

project\
project\setup.py
project\scripts\foo.py
project\scripts\bar.py
project\scripts\__init__.py
project\tests\test_foo.py
project\tests\__init__.py

My test file looks as follows

project\tests\test_fo.py

from ..scripts import foo

def test_one():
     assert 0

I get the following error, when I do

cd C:\project
C:\virtualenvs\test_env\Scripts\activate
python setup.py install
python setup.py test

E ValueError: Attempted relative import beyond toplevel package

What am I doing wrong? This is my setup.py

setup(
    name = 'project',
    setup_requires=['pytest-runner'],
    tests_require=['pytest'],
    packages = ["scripts","tests"],
    package_data={
          'scripts': ['*.py'],
          'tests': ['*.py'],
         },
)

Answer 1

Relative imports only work within a package. scripts may be a package, and so is tests, but the project directory is not (and neither should it be). This makes scripts and tests top-level packages. You can't refer to other top-level names using relative syntax.

Moreover, tests are not run with the tests package; the test runner imports the test_foo module, not the tests.test_foo module, so as far as Python is concerned test_foo is a top-level module.

scripts is a top-level name, just use that directly. You will have to add the project directory to sys.path however. You can do so at the top of your test_foo.py file with:

import os
import sys

TEST_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = os.path.abspath(os.path.join(TEST_DIR, os.pardir))
sys.path.insert(0, PROJECT_DIR)

then import from scripts with absolute paths:

from scripts import foo

Note however that when you run python setup.py then your current working directory is added to sys.path anyway, so scripts is available directly without having to fiddle with sys.path.

Moreover, pytest will already do the work for you; for any given test file it'll make sure the first parent directory with no __init__.py file in it is on sys.path. In your case that's the project/ directory, so again scripts is directly available to import from. See Good Practices:

If pytest finds a “a/b/test_module.py” test file while recursing into the filesystem it determines the import name as follows:

• determine basedir: this is the first “upward” (towards the root) directory not containing an __init__.py. If e.g. both a and b contain an __init__.py file then the parent directory of a will become the basedir.
• perform sys.path.insert(0, basedir) to make the test module importable under the fully qualified import name.
• import a.b.test_module where the path is determined by converting path separators / into ”.” characters. This means you must follow the convention of having directory and file names map directly to the import names.

Note that in order to actually use pytest to run your tests when you use setup.py test, you need to register an alias in your setup.cfg file (create it in project/ if you do not have one):

[aliases]
test = pytest