How to build and distribute a Python/Cython package that depends on third party libFoo.so

Question

I've written a Python module that depends on some C extensions. Those C extensions depend in turn on several compiled C libraries. I'd like to be able to distribute this module bundled with all the dependencies.

I've put together a minimal example (it can be found on GitHub in its entirety).

The directory structure is:

$ tree .
.
├── README.md
├── poc
│   ├── __init__.py
│   ├── cython_extensions
│   │   ├── __init__.py
│   │   ├── cvRoberts_dns.c
│   │   ├── cvRoberts_dns.h
│   │   ├── helloworld.c
│   │   ├── helloworld.pxd
│   │   ├── helloworld.pyx
│   │   ├── test.c
│   │   └── test.h
│   ├── do_stuff.c
│   └── do_stuff.pyx
└── setup.py

setup.py builds the extensions, and links against the necessary libraries (libsundials_cvode, libsundials_nvectorserial in this case):

from setuptools import setup, find_packages
from setuptools.extension import Extension
from Cython.Build import cythonize


ext_module_dostuff = Extension(
    'poc.do_stuff',
    ['poc/do_stuff.pyx'],
)

ext_module_helloworld = Extension(
    'poc.cython_extensions.helloworld',
    ['poc/cython_extensions/helloworld.pyx', 'poc/cython_extensions/test.c', 'poc/cython_extensions/cvRoberts_dns.c'],
    include_dirs = ['/usr/local/include'],
    libraries = ['m', 'sundials_cvodes', 'sundials_nvecserial'],
    library_dirs = ['/usr/local/lib'],
)

cython_ext_modules = [
   ext_module_dostuff,
   ext_module_helloworld
]


setup (
  name = "poc",
  ext_modules = cythonize(cython_ext_modules),
  packages=['poc', 'poc.cython_extensions'],
)

This is all well and good, but it does require that the end user first install sundials (and, in the actual case, several other libraries that are extremely finicky to get up and running).

Ideally, I'd like to be able to set this up only on development machines, create a distribution that includes the appropriate shared libraries, and ship some sort of bundle.

Given the various tutorials, examples and SO posts I've found so far. I'm led to believe I'm on the right track. However, there's some sort of final step that I'm just not groking.

Any help is appreciated :-).

Answer 1

As you probably know, the recommended way of distributing a Python module with compiled components is to use the wheel format. There doesn't appear to be any standard cross-platform way of bundling third-party native libraries into a wheel. However, there are platform-specific tools for this purpose.

On Linux, use auditwheel.

auditwheel modifies an existing Linux wheel file to add any third-party libraries which are not included in the basic "manylinux" standard. Here's an walkthrough of how to use it with your project on a clean install of Ubuntu 17.10:

First, install basic Python development tools, and the third-party library with its headers:

root@ubuntu-17:~# apt-get install cython python-pip unzip
root@ubuntu-17:~# apt-get install libsundials-serial-dev

Then build your project into a wheel file:

root@ubuntu-17:~# cd cython-example/
root@ubuntu-17:~/cython-example# python setup.py bdist_wheel
[...]
root@ubuntu-17:~/cython-example# cd dist/
root@ubuntu-17:~/cython-example/dist# ll
total 80
drwxr-xr-x 2 root root  4096 Nov  8 11:28 ./
drwxr-xr-x 7 root root  4096 Nov  8 11:28 ../
-rw-r--r-- 1 root root 70135 Nov  8 11:28 poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
root@ubuntu-17:~/cython-example/dist# unzip -l poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Archive:  poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
    62440  2017-11-08 11:28   poc/do_stuff.so
        2  2017-11-08 11:28   poc/__init__.py
   116648  2017-11-08 11:28   poc/cython_extensions/helloworld.so
        2  2017-11-08 11:28   poc/cython_extensions/__init__.py
       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst
      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json
        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt
      105  2017-11-08 11:28   poc-0.0.0.dist-info/WHEEL
      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA
      793  2017-11-08 11:28   poc-0.0.0.dist-info/RECORD
---------                     -------
   180382                     10 files

The wheel file can now be installed locally and tested:

root@ubuntu-17:~/cython-example/dist# pip install poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
hello cython
0.841470984808
trying to load the sundials program

3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
[...]

Now we install the auditwheel tool. It requires Python 3, but it's capable of processing wheels for Python 2 or 3.

root@ubuntu-17:~/cython-example/dist# apt-get install python3-pip
root@ubuntu-17:~/cython-example/dist# pip3 install auditwheel

auditwheel uses another tool called patchelf to do its job. Unfortunately, the version of patchelf included with Ubuntu 17.10 is missing a bugfix without which auditwheel will not work. So we'll have to build it from source (script taken from the manylinux Docker image):

root@ubuntu-17:~# apt-get install autoconf
root@ubuntu-17:~# PATCHELF_VERSION=6bfcafbba8d89e44f9ac9582493b4f27d9d8c369
root@ubuntu-17:~# curl -sL -o patchelf.tar.gz https://github.com/NixOS/patchelf/archive/$PATCHELF_VERSION.tar.gz
root@ubuntu-17:~# tar -xzf patchelf.tar.gz
root@ubuntu-17:~# (cd patchelf-$PATCHELF_VERSION && ./bootstrap.sh && ./configure && make && make install)

Now we can check which third-party libraries the wheel requires:

root@ubuntu-17:~/cython-example/dist# auditwheel show poc-0.0.0-cp27-cp27mu-linux_x86_64.whl

poc-0.0.0-cp27-cp27mu-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.4',
'GLIBC_2.2.5', 'GLIBC_2.3.4'}

The following external shared libraries are required by the wheel:
{
    "libblas.so.3": "/usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1",
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.26.so",
    "libgcc_s.so.1": "/lib/x86_64-linux-gnu/libgcc_s.so.1",
    "libgfortran.so.4": "/usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0",
    "liblapack.so.3": "/usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1",
    "libm.so.6": "/lib/x86_64-linux-gnu/libm-2.26.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.26.so",
    "libquadmath.so.0": "/usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0",
    "libsundials_cvodes.so.2": "/usr/lib/libsundials_cvodes.so.2.0.0",
    "libsundials_nvecserial.so.0": "/usr/lib/libsundials_nvecserial.so.0.0.2"
}

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libblas.so.3, libgfortran.so.4, liblapack.so.3, libquadmath.so.0,
libsundials_cvodes.so.2, libsundials_nvecserial.so.0

And create a new wheel which bundles them:

root@ubuntu-17:~/cython-example/dist# auditwheel repair poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Repairing poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/libsundials_nvecserial.so.0.0.2 -> poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2
Grafting: /usr/lib/libsundials_cvodes.so.2.0.0 -> poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0
Grafting: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1 -> poc/.libs/liblapack-549933c4.so.3.7.1
Grafting: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 -> poc/.libs/libblas-52fa99c8.so.3.7.1
Grafting: /usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0 -> poc/.libs/libgfortran-2df4b07d.so.4.0.0
Grafting: /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0 -> poc/.libs/libquadmath-0d7c3070.so.0.0.0
Setting RPATH: poc/cython_extensions/helloworld.so to "$ORIGIN/../.libs"
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
New WHEEL info tags: cp27-cp27mu-manylinux1_x86_64

Fixed-up wheel written to /root/cython-example/dist/wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
root@ubuntu-17:~/cython-example/dist# unzip -l wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
Archive:  wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA
        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt
       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst
      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json
     1400  2017-11-08 12:08   poc-0.0.0.dist-info/RECORD
      110  2017-11-08 12:08   poc-0.0.0.dist-info/WHEEL
    62440  2017-11-08 11:28   poc/do_stuff.so
        2  2017-11-08 11:28   poc/__init__.py
   131712  2017-11-08 12:08   poc/cython_extensions/helloworld.so
        2  2017-11-08 11:28   poc/cython_extensions/__init__.py
   230744  2017-11-08 12:08   poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0
  7005072  2017-11-08 12:08   poc/.libs/liblapack-549933c4.so.3.7.1
   264024  2017-11-08 12:08   poc/.libs/libquadmath-0d7c3070.so.0.0.0
  2039960  2017-11-08 12:08   poc/.libs/libgfortran-2df4b07d.so.4.0.0
    17736  2017-11-08 12:08   poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2
   452432  2017-11-08 12:08   poc/.libs/libblas-52fa99c8.so.3.7.1
---------                     -------
 10206026                     16 files

If we uninstall the third-party libraries, the previously-installed wheel will stop working:

root@ubuntu-17:~/cython-example/dist# apt-get remove libsundials-serial-dev && apt-get autoremove
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "poc/do_stuff.pyx", line 1, in init poc.do_stuff
ImportError: libsundials_cvodes.so.2: cannot open shared object file: No such file or directory

But the wheel with the bundled libraries will work fine:

root@ubuntu-17:~/cython-example/dist# pip uninstall poc
[...]
root@ubuntu-17:~/cython-example/dist# pip install wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
hello cython
0.841470984808
trying to load the sundials program

3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
[...]

On OSX, use delocate.

delocate for OSX apparently works very similarly to auditwheel. Unfortunately I don't have an OSX machine available to provide a walkthrough.

Combined example:

One project which uses both tools is SciPy. This repository, despite its name, contains the official SciPy build process for all platforms, not just Mac. Specifically, compare the Linux build script (which uses auditwheel), with the OSX build script (which uses delocate).

To see the result of this process, you might want to download and unzip some of the SciPy wheels from PyPI. For example, scipy-1.0.0-cp27-cp27m-manylinux1_x86_64.whl contains the following:

 38513408  2017-10-25 06:02   scipy/.libs/libopenblasp-r0-39a31c03.2.18.so
  1023960  2017-10-25 06:02   scipy/.libs/libgfortran-ed201abd.so.3.0.0

While scipy-1.0.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl contains this:

   273072  2017-10-25 07:03   scipy/.dylibs/libgcc_s.1.dylib
  1550456  2017-10-25 07:03   scipy/.dylibs/libgfortran.3.dylib
   279932  2017-10-25 07:03   scipy/.dylibs/libquadmath.0.dylib

Answer 2

To enhance mhsmith's excellent answer, here are the steps performed on MacOS with delocate:

1. Install sundials, for example with Homebrew:

   $ brew install sundials
   

2. Build the package:

   $ python setup.py bdist_wheel
   

3. The pendants to auditwheel show/auditwheel repair are delocate-listdeps/delocate-wheel, so first analyze the resulting wheel file:

   $ delocate-listdeps --all dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
   /usr/lib/libSystem.B.dylib
   /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_cvodes.2.9.0.dylib
   /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_nvecserial.2.7.0.dylib
   

4. Fixing the wheel file:

   $ delocate-wheel -v -w dist_fixed dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
   Fixing: dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
   Copied to package .dylibs directory:
     /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_cvodes.2.9.0.dylib
     /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_nvecserial.2.7.0.dylib
   

In the dist_fixed directory, you will have the bundled wheel. You will notice the size difference:

$ ls -l dist/ dist_fixed/
dist/:
total 72
-rw-r--r--  1 hoefling  wheel  36030 10 Nov 20:25 poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl

dist_fixed/:
total 240
-rw-r--r--  1 hoefling  wheel  120101 10 Nov 20:34 poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl

If you list deps for the bundled wheel, you will notice the needed libraries are now bundled (indicated by prefix @loader_path):

$ delocate-listdeps --all dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
/usr/lib/libSystem.B.dylib
@loader_path/../.dylibs/libsundials_cvodes.2.9.0.dylib
@loader_path/../.dylibs/libsundials_nvecserial.2.7.0.dylib

Installing the bundled wheel (notice the bundled libs are installed correctly):

$ pip install dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
Processing ./dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
Installing collected packages: poc
Successfully installed poc-0.0.0
$ pip show -f poc
Name: poc
Version: 0.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /Users/hoefling/.virtualenvs/stackoverflow-py27/lib/python2.7/site-packages
Requires: 
Files:
  poc-0.0.0.dist-info/DESCRIPTION.rst
  poc-0.0.0.dist-info/INSTALLER
  poc-0.0.0.dist-info/METADATA
  poc-0.0.0.dist-info/RECORD
  poc-0.0.0.dist-info/WHEEL
  poc-0.0.0.dist-info/metadata.json
  poc-0.0.0.dist-info/top_level.txt
  poc/.dylibs/libsundials_cvodes.2.9.0.dylib
  poc/.dylibs/libsundials_nvecserial.2.7.0.dylib
  poc/__init__.py
  poc/__init__.pyc
  poc/cython_extensions/__init__.py
  poc/cython_extensions/__init__.pyc
  poc/cython_extensions/helloworld.so
  poc/do_stuff.so