How to `pip install` a package that has non-Python dependencies?

Question

Many python packages have build dependencies on non-Python packages. I'm specifically thinking of lxml and cffi, but this dilemma applies to a lot of packages on PyPI. Both of these packages have unadvertised build dependencies on non-Python packages like libxml2-dev, libxslt-dev, zlib1g-dev, and libffi-dev. The websites for lxml and cffi declare some of these dependencies, but it appears that there is no way to do figure this out from a command line.

As a result, there are hundreds of questions on SO that take this general form:

pip install foo fails with an error: "fatal error: bar.h: No such file or directory". How do I fix it?

Is this a misuse of pip or is this how it is intended to work? Is there a sane way to know what build dependencies to install before running pip? My current approach is:

1. I want to install a package called foo.
2. pip install foo
3. foo has a dependency on a Python package bar.
   • If bar build fails, then look at error message and guess/google what non-Python dependency I need to install.
   • sudo apt-get install libbaz-dev
   • sudo pip install bar
   • Repeat until bar succeeds.
4. sudo pip uninstall foo
5. Repeat entire process until no error messages.

Step #4 is particularly annoying. Apparently pip (version 1.5.4) installs the requested package first, before any dependencies. So if any dependencies fail, you can't just ask pip to install it again, because it thinks its already installed. There's also no option to install just the dependencies, so you must uninstall the package and then reinstall it.

Is there some more intelligent process for using pip?

Answer 1

This is actually a comment about the answer suggesting using apt-get but I don't have enough reputation points to leave one.

If you use virtualenv a lot, then installing the python-packages through apt-get can become a pain, as you can get mysterious errors when the python packages installed system-wide and the python packages installed in your virtualenv try to interact with each other. One thing that I have found that does help is to use the build-dep feature. To build the matplotlib dependencies, for example:

sudo apt-get build-dep python-matplotlib 

And then activate your virtual environment and do pip install matplotlib. It will still go through the build process but many of the dependencies will be taken care of for you.

This is sort what the cran repositories suggest when installing R packages in ubuntu.

Answer 2

For most popular packages, There is a workaround for recent ubuntu systems. For example, I want to install matplotlib. When you order pip install matplotlib, it usually fails because of a missing dependency.

You can use apt-get install python-matplotlib instead. For python3, you can use apt-get install python3-matplotlib