How to declare build-time dependencies without breaking other packages?

Question

I ran into a problem when installing a package which depended on python-daemon. I ultimately traced it to the latest version of the package python-daemon (2.0.3) released yesterday. Testing in a virtual environment on an Ubuntu 14.04 machine and issuing the following commands:

(venv) $ pip list
argparse (1.2.1)
pip (1.5.6)
setuptools (3.6)
wsgiref (0.1.2)
(venv) $ pip install redis
 ... works fine ....
(venv) $ pip install python-daemon
 ...
 snip
 ...   
 File "/home/pwj/.virtualenvs/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load

['__name__'])

ImportError: No module named version

(venv)02:15 PM tmp$ pip list
argparse (1.2.1)
lockfile (0.10.2)
pip (1.5.6)
python-daemon (2.0.3)
setuptools (3.6)
wsgiref (0.1.2)

So the install of python-daemon seemed to work but something affected pip or setuptools because other packages (celery, flask), I try to install with pip after this gives me the same traceback:

 ...
 snip
 ...   

 File "/home/pwj/.virtualenvs/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load

['__name__'])

ImportError: No module named version

If I uninstall python-daemon with pip things again and packages that weren't installing now install fine. Has anyone else come across this or something similar with a different project? My solution was to pip install the previous version

(venv) $ pip install python-daemon==2.0.2
... works ...

but was wondering what might be causing such an error.

Answer 1

(This behaviour is corrected in python-daemon version 2.0.4 and later.)

There are two sides to this:

• Setuptools assumes it is the centre of everything.
• Version 2.0.3 of python-daemon doesn't take that into account.

A more detailed explanation: There is some complex code using Docutils involved in the python-daemon build process, that isn't needed after install and isn't part of the library code.

It's too complex to leave in the un-importable (and therefore not-unit-testable) setup.py, so that build code is shunted to a separate testable module, version (in the file version.py), which itself uses Docutils.

But then the setup.py has a circular dependency: How to import version, when Docutils isn't yet installed? How to use Setuptools to ensure Docutils is installed, when running setup.py to completion will need version? All the feasible solutions are ugly and confusing.

The approach taken in ‘python-daemon’ 2.0.3 is to declare Docutils required for setup, and declare a Setuptools entry point for the work that needs version. That way setup.py gets to install Docutils before any of the entry points that will use version.

But now we come to the first point, that Setuptools arrogates itself as the centre of everything. By declaring an entry point, setup.py has modified every Setuptools action thereafter, and every package will fail if it can't find the entry points. And, since most of them don't have version or the specified functions in that module, they crash Setuptools.

What is essentially a bug to be fixed, reveals a poorly-understood corner case in Setuptools. So I'm voting your question up.

There doesn't seem to be a good solution to this: having modules available for setup.py but ensuring requirements are met first. Setuptools assumes it is the only build system needed to satisfy all dependencies for everything, and when that assumption fails it's very difficult to get around.

Thanks to the Python Packaging Authority folks, and the distutils-sig forum, for explaining this to me.