Is it possible to trick pip install --find-links into using a downloaded sdist for --editable requirements?

Question

Using the following command:

pip install -r requirements.txt -d sdists/

You can easily create an archive of requirements for distributing with your project. This works great if your requirements look like this:

Django==1.3.1
django-tagging==0.3.1
django-robots==0.6.1

You can then install those requirements without touching PyPI at all, like so:

pip install -r requirements.txt --find-links sdists/ --no-index

Is it possible to use the same method for --editable requirements? E.g.:

-e hg+https://bitbucket.org/ubernostrum/django-contact-form/@1d3791fa4dfb#egg=django-contact-form

As far as I can tell, pip install -d happily downloads editable requirements and creates an sdist for you, but pip install --find-links does not have any way to match up the downloaded sdist with the associated line in your requirements file, so it ignores the downloaded sdist and continues checking out the code from VCS as usual.

Answer 1

While it doesn't appear that this is strictly possible using PIP, there is a workaround that accomplishes the same thing. The workaround is to automatically generate a second requirements file from the original requirements file and sdists directory (to be used only for that directory).

A simple implementation might look something like this (save in a file called "make_reqs.py"):

#!/usr/bin/env python

import re
import sys
import os.path

pat = '.+#egg=(.+)'
allowed_exts = ['.zip', '.tar.gz', 'tar.bz2']

def find_version(sdists_dir, name):
    files = [f for f in os.listdir(sdists_dir) if f.startswith(name)]
    if len(files) != 1:
        return ''
    version = files[0].replace(name+'-', '')
    for ext in allowed_exts:
        version = version.replace(ext, '')
    return version

def get_requirements(file_name, sdists_dir):
    out_reqs = ['--find-links file://%s' % os.path.abspath(sdists_dir)]
    with open(file_name) as req_file:
        reqs = [l.strip() for l in req_file.readlines()]
        for req in reqs:
            match = re.match(pat, req)
            if match and not req.startswith('#'):
                name = match.group(1)
                version = find_version(sdists_dir, name)
                if version:
                    out_reqs.append('%s==%s' % (name, version))
                else:
                    out_reqs.append(req)
            else:
                out_reqs.append(req)
    return out_reqs

if __name__ == '__main__':
    print '\n'.join(get_requirements(*sys.argv[1:]))

To use the script, you would do something like this:

python make_reqs.py requirements.txt /path/to/sdists > sdist_reqs.txt
pip install --no-index -r sdist_reqs.txt