Building a ctypes-"based" C library with distutils

Question

Following this recommendation, I have written a native C extension library to optimise part of a Python module via ctypes. I chose ctypes over writing a CPython-native library because it was quicker and easier (just a few functions with all tight loops inside).

I've now hit a snag. If I want my work to be easily installable using distutils using python setup.py install, then distutils needs to be able to build my shared library and install it (presumably into /usr/lib/myproject). However, this not a Python extension module, and so as far as I can tell, distutils cannot do this.

I've found a few references to people other people with this problem:

• Someone on numpy-discussion with a hack back in 2006.
• Somebody asking on distutils-sig and not getting an answer.
• Somebody asking on the main python list and being pointed to the innards of an existing project.

I am aware that I can do something native and not use distutils for the shared library, or indeed use my distribution's packaging system. My concern is that this will limit usability as not everyone will be able to install it easily.

So my question is: what is the current best way of distributing a shared library with distutils that will be used by ctypes but otherwise is OS-native and not a Python extension module?

Feel free to answer with one of the hacks linked to above if you can expand on it and justify why that is the best way. If there is nothing better, at least all the information will be in one place.

Answer 1

The distutils documentation here states that:

A C extension for CPython is a shared library (e.g. a .so file on Linux, .pyd on Windows), which exports an initialization function.

So the only difference regarding a plain shared library seems to be the initialization function (besides a sensible file naming convention I don't think you have any problem with). Now, if you take a look at distutils.command.build_ext you will see it defines a get_export_symbols() method that:

Return the list of symbols that a shared extension has to export. This either uses 'ext.export_symbols' or, if it's not provided, "PyInit_" + module_name. Only relevant on Windows, where the .pyd file (DLL) must export the module "PyInit_" function.

So using it for plain shared libraries should work out-of-the-box except in Windows. But it's easy to also fix that. The return value of get_export_symbols() is passed to distutils.ccompiler.CCompiler.link(), which documentation states:

'export_symbols' is a list of symbols that the shared library will export. (This appears to be relevant only on Windows.)

So not adding the initialization function to the export symbols will do the trick. For that you just need to trivially override build_ext.get_export_symbols().

Also, you might want to simplify the module name. Here is a complete example of a build_ext subclass that can build ctypes modules as well as extension modules:

from distutils.core import setup, Extension from distutils.command.build_ext import build_ext   class build_ext(build_ext):      def build_extension(self, ext):         self._ctypes = isinstance(ext, CTypes)         return super().build_extension(ext)      def get_export_symbols(self, ext):         if self._ctypes:             return ext.export_symbols         return super().get_export_symbols(ext)      def get_ext_filename(self, ext_name):         if self._ctypes:             return ext_name + '.so'         return super().get_ext_filename(ext_name)   class CTypes(Extension): pass   setup(name='testct', version='1.0',       ext_modules=[CTypes('ct', sources=['testct/ct.c']),                    Extension('ext', sources=['testct/ext.c'])],       cmdclass={'build_ext': build_ext})