How to install python binding of a C++ library

Question

Imaging that we are given a finished C++ source code of a library, called MyAwesomeLib. The goal is to expose some of its power to python, so we create a wrapper using swig and generated a python package called PyMyAwesomeLib.

The directory structure now looks like

root_dir
|-src/
|-lib/
|    |- libMyAwesomeLib.so
|    |- _PyMyAwesomeLib.so
|-swig/
|    |- PyMyAwesomeLib.py
|-python/
     |- Script_using_myawesomelib.py

So far so good. Ideally, all we want to do next is to copy lib/*.so swig/*.py and python/*.py into the corresponding directory in site-packages in a pythonic way, i.e. using

python setup.py install

However, I got very confused when trying to achieve this simple goal using setuptools and distutils. Both tools handles the compilation of python extensions through an internal system, where the source file, compiler flags etc. are passed using setup(ext_module=[Extension(...)]). But this is ridiculous since MyAsesomeLib has a fully functioning build system that is based on makefile. Porting the logic embedded in makefiles would be redundant and completely un-necessary work.

After some research, it seems there are two options left, I can either override setuptools.command.build and setuptools.command.install to use the existing makefile and copy the results directly, or I can somehow let setuptools know about these files and ask it to copy them during installation. The second way is more appealing, but it is what gives me the most headache. I have tried the following optionts without success

• package_data, and include_package_data does not work because *.so files are not under version control and they are not inside of any package.
• data_files does not seems to work since the files only get included when running python setup.py sdist, but ignored when python setup.py install. This is the opposite of what I want. The .so files should not be included in the source distribution, but get copied during the installation step.
• MANIFEST.in failed for the same reason as data_files.
• eager_resources does not work either, but honestly I do not know the difference between eager_resources and data_files or MANIFEST.in.

I think this is actually a common situation, and I hope there is a simple solution to it. Any help would be greatly appreciated.

Answer 1

Porting the logic embedded in makefiles would be redundant and completely un-necessary work.

Unfortunately, that's exactly what I had to do. I've been struggling with this same issue for a while now.

Porting it over actually wasn't too bad. distutils does understand SWIG extensions, but it this was implemented rather haphazardly on their part. Running SWIG creates Python files, and the current build order assumes that all Python files have been accounted for before running build_ext. That one wasn't too hard to fix, but it's annoying that they would claim to support SWIG without mentioning this. Distutils attempts to be cross-platform when compiling things, so there is still an advantage to using it.

If you don't want to port your entire build system over, use the system's package manager. Many complex libraries do this (but they also try their best with setup.py). For example, to get numpy and lxml on Ubuntu you'd just do: sudo apt-get install python-numpy python-lxml. No pip.

I realize you'd rather write one setup file instead of dealing with every package manager ever so this is probably not very helpful.

If you do try to go the setuptools route there is one fatal flaw I ran into: dependencies.

For instance, if you are distributing a SWIG-based project, it's going to need libpython. If they don't have it, an error like this happens:

#include <Python.h>
error: File not found

That's pretty unhelpful to the average user.

Even worse, if you require a shared library but the user's library is out of date, the user can get some crazy errors. You're at the mercy of their C++ compiler to output Google-friendly error messages so they can figure it out.

The long-term solution would be to get setuptools/distutils to get better at detecting non-python libraries, hopefully as good as Ruby's gem. I pretty much had to roll my own. For instance, in this setup.py I'm working on you can see a few functions at the top I hacked together for dependency detection (still doesn't work on all systems...definitely not Windows).