How to avoid building C library with my python package?

Question

I'm building a python package using a C library with ctypes. I want to make my package portable (Windows, Mac and Linux).

I found a strategy, using build_ext with pip to build the library during the installation of my package. It creates libfoo.dll or libfoo.dylib or libfoo.so depending on the target's platform.

The problem with this is that my user needs CMake installed.

Does exist another strategy to avoid building during the installation? Do I have to bundle built libraries in my package?

I want to keep my users doing pip install mylib.

Edit: thank to @Dawid comment, I'm trying to make a python wheel with the command python setup.py bdist_wheel without any success.

How can I create my python wheel for different platform with the embedded library ?

Edit 2: I'm using python 3.4 and working on Mac OS X, but I have access to Windows computer, and Linux computer

Answer 1

You're certainly heading down the right path according to my research... As Daniel says, the only option you have is to build and distribute the binaries yourself.

In general, the recommended way to install packages is covered well in the packaging user guide. I won't repeat advice there as you have clearly already found it. However the key point in there is that the Python community, specifically PyPA are trying to standardize on using platform wheels to package binary extensions. Sadly, there are a few issues at this point:

1. You cannot create distributions for all Linux variants, but you can build wheels for a compatible subset. See https://www.python.org/dev/peps/pep-0513/ for details.
2. The advice on building extensions is somewhat incomplete, reflecting the lack of a complete solution for binary distributions.
3. People then try to build their own library and distribute it as a data file, which confuses setuptools.

I think you are hitting this last issue. A workaround is to force the Distribution to build a platform wheel by overriding is_pure() to always return False. However you could just keep your original build instructions and bdist_wheel should handle it.

Once you've built the wheel, though, you still need to distribute it and maybe other binary packages that it uses or use it. At this point, you probably need to use one of the recommended tools like conda or a PyPI proxy like devpi to serve up your wheels.

EDIT: To answer the extra question about cross-compiling

As covered here Python 2.6 and later allows cross-compilation for Windows 32/64-bit builds. There is no formal support for other packages on other platforms and people have had limited success trying to do it. You are really best off building natively on each of your Linux/Mac/Windows environments.

Answer 2

@rth and @PeterBrittain help me a lot. Here the solution I use:

structure folder :

setup.py
python_package/
    lib/
        libfoo.dylib
        libfoo.dll
    __init__.py
    main.py

setup.py :

from setuptools import setup, dist


class BinaryDistribution(dist.Distribution):
    def is_pure(self):
        return False


setup(
    name='python_package',
    package_data={'python_package': ['lib/libfoo.dylib','lib/libfoo.dll']},
    include_package_data=True,
    distclass=BinaryDistribution,
    packages=['python_package'],
)

main.py :

#!/usr/bin/env python
import platform
from ctypes import CDLL, c_char_p

import pkg_resources

sysname = platform.system()

if sysname == 'Darwin':
    lib_name = "libfoo.dylib"
elif sysname == 'Windows':
    lib_name = "libfoo.dll"
else:
    lib_name = "libfoo.so"
lib_path = pkg_resources.resource_filename('python_package', 'lib/{}'.format(lib_name))
foo = CDLL(lib_path)

bar = foo.bar
bar.restype = c_char_p
bar.argtypes = [c_char_p]

print(bar('hello'))

build wheel :

python setup.py bdist_wheel

It creates a specific plateform wheel rtfdoc-0.0.1-cp34-cp34m-macosx_10_10_x86_64.whl and mac users can do a simple pip install rtfdoc-0.0.1-cp34-cp34m-macosx_10_10_x86_64.whl

This solution is not entirely satisfactory:

• I think I will still need to create a Extension or look at SCons for linux users.
• Update my package will be difficult because of the production of the lib
• I don't know how to manage .dll for 32 bit and 64 bit

Thank you, I learned a lot, and I understand why setup.py in Pillow or Psycopg2 are huge