Installing PIP packages to a Virtualenv using a download cache

Question

What's the proper way to install pip packages to a virtualenv using cached packages? I've tried setting --timeout=360 and --use-mirrors, but pypi performance is so flakey, it routinely hangs or disconnects, making deployments a crapshoot.

So, my plan was to use the --download-cache option to pre-download all package archives (but not install them), e.g.:

pip install --upgrade --timeout=120 --use-mirrors --no-install --download-cache /usr/local/pip/cache -r pip-requirements.txt

and then specify this cache location during the install into the virtualenv, e.g.:

virtualenv /usr/local/myapp/.env
. /usr/local/myapp/.env/bin/activate; pip install --no-download --download-cache /usr/local/pip/cache -r pip-requirements.txt
deactivate

Creating the cache was very frustrating, because pip seems to nondeterministically save the downloaded package to the cache dir. Specifically, it refused to cache Django==1.4.0, saying "that requirement has already been met", so I had to use the --force-reinstall.

Installing using the cache is currently where I'm stuck. Running the above command gives me the error:

Could not install requirement Django==1.4.0 (from -r pip-requirements.txt (line 1)) because source folder /usr/local/myapp/.env/build/Django does not exist (perhaps --no-download was used without first running an equivalent install with --no-install?)
Storing complete log in /home/chris/.pip/pip.log

What does this mean?

The files:

http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FD%2FDjango%2FDjango-1.4.tar.gz
http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FD%2FDjango%2FDjango-1.4.tar.gz.content-type

definitely exist in the cache directory, and I used the --no-install option.

What am I doing wrong?

Answer 1

The problem seems to be that --download-cache can only be used to specify the cache for downloading, not for installing. Therefore pip is still looking at /usr/local/myapp/.env/build/Django instead of /usr/local/pip/cache. Have you tried moving

pip install --upgrade --timeout=120 --use-mirrors --no-install --download-cache /usr/local/pip/cache -r pip-requirements.txt

to after the creation of virtualenv? I wonder if that would help.

You may also want to try to export PIP_DOWNLOAD_CACHE and see if it works without using --download-cache.

Here is what I find that works:

pip install --no-install --use-mirrors --download=/DIRNAME PKGNAME
pip install --find-links=file:///DIRNAME --no-index --index-url=file:///dev/null PKGNAME

Now, actually, here is the tool for I would use instead of all the above and it solves all of the problems much more elegantly and reliably: pip2pi by David Wolever.

From the docs:

pip2pi builds a PyPI-compatible package repository from pip requirements

pip2pi allows you to create your own PyPI index by using two simple commands:

1. To mirror a package and all of its requirements, use pip2tgz:

   $ cd /tmp/; mkdir package/
   $ pip2tgz packages/ httpie==0.2
   ...
   $ ls packages/
   Pygments-1.5.tar.gz
   httpie-0.2.0.tar.gz
   requests-0.14.0.tar.gz
   

2. To build a package index from the previous directory:

   $ ls packages/
   bar-0.8.tar.gz
   baz-0.3.tar.gz
   foo-1.2.tar.gz
   $ dir2pi packages/
   $ find packages/
   /httpie-0.2.0.tar.gz
   /Pygments-1.5.tar.gz
   /requests-0.14.0.tar.gz
   /simple
   /simple/httpie
   /simple/httpie/httpie-0.2.0.tar.gz
   /simple/Pygments
   /simple/Pygments/Pygments-1.5.tar.gz
   /simple/requests
   /simple/requests/requests-0.14.0.tar.gz
   

3. To install from the index you built in step 2., you can simply use:

   pip install --index-url=file:///tmp/packages/simple/ httpie==0.2
   

Bonus: you can even mirror your own index to a remote host with pip2pi.