How to include a shared C library in a Python package

Question

I have a project depending on a shared library. To make it clear from the beginning: the shared library is a pure C library and not a Python library. For reasons of simplicity I created a small demo project called pkgtest which I will refer to.

So what needs to be done is: Run a Makefile to compile the library and place the compiled shared library (called libhello.so here) file somewhere it can be accessed from within the depending Python package.

My best guess so far was to run the makefile as a preinstallation routine, copy the libhello.so file in the packages directory and add it to the package_data parameter of the setup script. When installed the shared library then gets placed in the site-packages/pkgtest/ directory and can be accessed from the module.

The package directory is structure is as simple as this:

pkgtest/
  src/
     libhello.c
     libhello.h
     Makefile
  pkgtest/
    __init__.py
    hello.py
  setup.py

My setup.py looks like this:

setup.py

import subprocess
from setuptools import setup
from distutils.command.install import install as _install


class install(_install):
    def run(self):
        subprocess.call(['make', 'clean', '-C', 'src'])
        subprocess.call(['make', '-C', 'src'])
        _install.run(self)


setup(
    name='pkgtest',
    version='0.0.1',
    author='stefan',
    packages=['pkgtest'],
    package_data={'pkgtest': ['libhello.so']},
    cmdclass={'install': install},
)

The Makefile actually builds the library and copies it into the directory of my python package.

src/Makefile

all: libhello.so

libhello.o: libhello.c
        gcc  -fPIC -Wall -g -c libhello.c

libhello.so: libhello.o
        gcc -shared -fPIC -o libhello.so libhello.o
        cp libhello.so ../pkgtest/libhello.so

clean:
        rm -f *.o *.so

So all hello.py is actually doing is load the library and call the hello function that prints some text. But for completeness I will show the code here:

pkgtest/hello.py

import os
import ctypes

basedir = os.path.abspath(os.path.dirname(__file__))
libpath = os.path.join(basedir, 'libhello.so')

dll = ctypes.CDLL(libpath)

def say_hello():
    dll.hello()

So this actually works but what I don't like about this approach is that the shared library lives in the directory of the Python package. I figure it would be better to put it in some sort of central library directory such as /usr/lib/. But for this one would need root privileges on installation. Has somebody got some experience with this kind of problem and would like to share a solution or helpful idea. Would be great.

Answer 1

You can create a Python package which includes shared libraries and works on (almost) any linux distro using manylinux.

The goal of the manylinux project is to provide a convenient way to distribute binary Python extensions as wheels on Linux. This effort has produced PEP 513 which defines the manylinux1_x86_64 and manylinux1_i686 platform tags.

The general procedure is:

1. Build the external library and the Python package inside one of the docker containers provided by the manylinux team (see python-manylinux-demo)
2. Run auditwheel repair to copy the external shared libraries that your package depends on into the Python wheel, setting the RPATH accordingly.

See .travis.yml and build-wheels.sh in the python-manylinux-demo repo for an example.

Answer 2

package_data is for data. setup can do the compilation into an *.so.

Following my solution in python setup.py build ctypes.CDLL: cannot open shared object file: No such file or directory, I think your setup could employ ext_modules and py_modules, something like

setup(name='pkgtest',
      py_modules=['pkgtest'],
      ext_modules=[Extension('src.libhello', ['src/libhello.c'])]
     )