pip fails to install packages from requirements.txt

Question

I am trying to install a python software using the requirements file.

>> cat requirements.txt
Cython==0.15.1
numpy==1.6.1
distribute==0.6.24
logilab-astng==0.23.1logilab-common==0.57.1
netaddr==0.7.6
numexpr==2.0.1
ply==2.5
pycallgraph==0.5.1
pyflowtools==0.3.4.1
pylint==0.25.1
tables==2.3.1
wsgiref==0.1.2

So I create a virtual environment

>> mkvirtualenv parser

(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2

(parser)
>> pip install -r requirements.txt

... and then I packages downloaded but not installed with errors: http://pastie.org/4079800

(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2

Surprisingly, if I try to manually install each package, they install just fine. For instance:

>> pip install numpy==1.6.1

(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2
numpy==1.6.1

I am lost. What is going on?

PS: I am using pip v1.1 and python v2.7.2 with virtualenv and virtualenvwrapper

Answer 1

It looks like the numexpr package has an install-time dependency on numpy. Pip makes two passes through your requirements: first it downloads all packages and runs each one's setup.py to get its metadata, and then it installs them all in a second pass.

So, numexpr is trying to import from numpy in its setup.py, but when pip first runs numexpr's setup.py, it has not yet installed numpy.

This is also why you don't see this error when you install the packages one by one: if you install them one at a time, numpy will be fully installed in your environment before you pip install numexpr.

The only solution is to install pip install numpy before you ever run pip install -r requirements.txt -- you won't be able to do this in a single command with a single requirements.txt file.

More info here: https://github.com/pypa/pip/issues/25

Answer 2

I come across with a similar issue and I ended up with the below:

cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 python -m pip install 

That will read line by line the requirements.txt and execute pip. I cannot find from where I got the answer properly, so apologies for that, but I found some justification below:

1. How sed works: https://howto.lintel.in/truncate-empty-lines-using-sed/
2. Another similar answer but with git: https://stackoverflow.com/a/46494462/7127519

Hope this help with alternatives.