setuptools and pip: choice of minimal and complete install

Question

We've made a library which depends on other libraries. But there are necessary (e.g. for server batch processing) and optional dependencies (e.g. for clients with GUI).

Is something like this possible:

pip install mylib.tar.gz  # automatically downloads and installs with the minimal set of dependencies

pip install mylib.tar.gz  --install-option="complete"  # automatically installs with all dependencies

I've found the extra_require flag, but how can I tell pip to use them? The setup.py looks like this:

from setuptools import setup

# ...

# Hard library depencencies:
requires = [
    "numpy>=1.4.1",
    "scipy>=0.7.2",
    "traits>=3.4.0"
]

# Soft library dependencies:
recommended = {
    "mpl": ["matplotlib>=0.99.3"],
    "bn": ["bottleneck>=0.6"]
}

# ...

# Installer parameters:
setup(
    name = "mylib",
    #...
    install_requires = requires,
    extras_require = recommended
)

Answer 1

You can install the packages in extras_require by appending the name of the recommended dependency in square brackets (i.e. [mpl] or [bn] in your case) to the package name in pip.

So to install 'mylib' with the additional requirements, you would call pip like this:

pip install 'mylib[mpl]'
pip install 'mylib[bn]'

This will first download and install the extra dependencies, and then mylib's core dependencies.

This is anologous to how you declare those dependencies with setuptools: http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies (see the install_requires value in the third example)

Answer 2

So pip is actually quite picky about installing libraries with extra requirements

pip install -e ".[extra,requirements]"    # works with file paths
pip install "package[extra,requirements]" # works when downloading packages
pip install ".[extra,requirments]"        # DOES NOT WORK

I think this is down to how the RequirementsSpec parser works, and pip does some extra magic with the -e flag. Anyhow after much head banging, here's a mildly ugly workaround

pip install "file:///path/to/your/python_code#egg=SomeName[extra,requirements]"

The egg=SomeName part is basically ignored, but pip correctly picks up the extra requirements

Caveats

• Tested with pip 1.5.6 so make sure you're using a current version of pip.
• As far as I can tell, the file:/// syntax is undocumented in pip, so I'm not sure if it'll change in the future. It looks a bit like the VCS Support syntax but I was a bit surprised it worked.
• You could also get around this by running your own pypi server, but that's a bit out of scope.