Tests and python package structure

Question

I have some problems in structuring my python project. Currently it is a bunch of files in the same folder. I have tried to structure it like

proj/
  __init__.py
  foo.py
  ...
  bar/
    __init__.py
    foobar.py
    ...
  tests/
    foo_test.py
    foobar_test.py
    ...

The problem is that I'm not able, from inner directories, to import modules from the outer directories. This is particularly annoying with tests.

I have read PEP 328 about relative imports and PEP 366 about relative imports from the main module. But both these methods require the base package to be in my PYTHONPATH. Indeed I obtain the following error

ValueError: Attempted relative import in non-package.

So I added the following boilerplate code on top of the test files

import os, sys
sys.path.append(os.path.join(os.getcwd(), os.path.pardir))

Still I get the same error. What is the correct way to

• structure a package, complete with tests, and
• add the base directory to the path to allow imports?

EDIT As requested in the comment, I add an example import that fails (in the file foo_test.py)

import os, sys
sys.path.append(os.path.join(os.getcwd(), os.path.pardir))
from ..foo import Foo

Answer 1

When you use the -m switch to run code, the current directory is added to sys.path. So the easiest way to run your tests is from the parent directory of proj, using the command:

python -m proj.tests.foo_test

To make that work, you will need to include an __init__.py file in your tests directory so that the tests are correctly recognised as part of the package.

Answer 2

I like to import modules using the full proj.NAME package prefix whenever possible. This is the approach the Google Python styleguide recommends.

One option to allow you to keep your package structure, use full package paths, and still move forward with development would be to use a virtualenv and put your project in develop mode. Your project's setup.py will need to use setuptools instead of distutils, to get the develop command.

This will let you avoid the sys.path.append stuff above:

% virtualenv ~/virt
% . ~/virt/bin/activate
(virt)~% cd ~/myproject
(virt)~/myproject% python setup.py develop
(virt)~/myproject% python tests/foo_test.py

Where foo_test.py uses:

from proj.foo import Foo

Now when you run python from within your virtualenv your PYTHONPATH will point to all of the packages in your project. You can create a shorter shell alias to enter your virtualenv without having to type . ~/virt/bin/activate every time.