Building/testing a Python project with C extensions

Question

I have a project with a python package and a compiled component inside of it. Current directory layout is:

<project>
  foo/
  foo/__init__.py
  foo/...

  src/
  src/c_foo.c

  tests/
  tests/test_foo.py

  setup.py

When the project is built, distutils create a build/lib directory which I either add to PYTHONPATH or install into a virtual environment. Resulting structure is the following:

<project>
  build/lib
  build/lib/foo/__init__.py
  build/lib/foo/c_foo.so

The issue with that is that if I start a python interpreter session from the project root, run tests from the project root etc., it picks up the source tree instead of the built tree.

I found several existing solutions to this in use:

1. Place python sources under a separate directory, eg. lib/foo, modules/foo etc. Downside to that is an extra directory level to all of the source files and inconsistency with projects that don't have compiled extensions and therefore have their python packages in the root directory.

2. Keep the packages in the root, which means an inconvenience having to chdir out of the project root (eg. into tests/ directory) so that python interpreter does not see the source packages (via build script or manually).

3. Keep the packages in the root under a different name (eg. foo-module or foo-lib) with an appropriate package_dir={'foo':'lib-foo'} line in setup.py. This is a variation of pt. 1 without an extra level of directory hierarchy, which is pretty much the same thing, I suppose.

4. Keep the packages in the root and use setup.py build_ext --inplace, however this contaminates the source tree.

Either case introduces an overhead vs. a plain python project where one can modify/run code right out of the source tree. I'd very much like to hear everyone's thoughts on pros/cons of the above and which particular method you use for you projects.

Answer 1

You may want to try the develop target in distribute (formerly setuptools).

Make sure distribute is installed, then modify your setup.py like so:

# the setuptools package name is still used
from setuptools import setup, Extension
...

Then enter your virtualenv and run develop:

% source ~/virt/bin/activate
(virt)% cd ~/project
(virt)% python setup.py develop

You should be able to run your tests from within the project root, and anytime you activate that virtualenv you can access that project's packages and extension regardless of your path:

% cd /tmp
% source ~/virt/bin/activate
(virt)% python -c 'import foo, c_foo; print foo, c_foo'

<module 'foo' from '/Users/user/project/foo/__init__.py'>
<module 'c_foo' from '/Users/user/project/c_foo.so'>