Possible to autogenerate Cython bindings around a large, existing C library?

Question

In otherwords: *.h/*.c --[??POSSIBLE??]--> *.pxd/*.pyx

OK. I’ve done (I hope) enough digging around the Internet - but I think this is a good question so I’ll ask it straight.

There are a few related questions (e.g. Generate python bindings, what methods/programs to use or Wrapping a C library in Python: C, Cython or ctypes? ) but which don't quite sum up the situation that I’m asking which is perhaps for a more “high-level” approach (and specifically for an existing library, not generating new C from python).

I’ve got a little bit of experience of this myself having wrapped a wee bit of code before using Cython. Cython gets the thumbs up for speed and maintainability. That’s OK in my book for small/single bits of code - but, this time I’ve got a bit more on my plate…

And following the first of the three great virtues of a programmer - I want to do this with as minimal effort as possible.

So the real question here is how can I ease the creation by automated means of the .pxd, and possibly .pyx, files (i.e. to save time and not slip up miss-typing something).

This here seems to be the only real hint/note about how to do this - but most of the projects on it are defunct, old or sourceforge. Many only seem to work for C++ (this is C I'm doing here).

Does anyone still use them? Recently? Has anyone got a workflow or best practice for doing this? Am I simply just better doing it by hand?

My library is well defined by a set of header files. One containing defs of all the C struct/types and another containing prototypes for all the functions. But it's loooonnnggg...

Thanks for any tips.

UPDATE (25th August, 2015):

Right, so over the last few months when I had a spare moment, I tried:

• CFFI (thank for @David pointing that out) - has a noble aim of "to call C code from Python without learning a 3rd language: existing alternatives require users to learn domain specific language (Cython, SWIG) or API (ctypes)” - but it didn’t quite fit the bill as it involved a fair degree of embedded C code in the actual python files (or loading it in). This would be a pretty manual process to do for a large library. Maybe I missed something…

• SWIG is the granddaddy of Python binding, and is pretty solid. Fundamentally though, it is not “hands off” as I understand it - i.e. you need a separate specification file. For example, you have to edit all your C header files to indicate building a python module with a #define SWIG_FILE_WITH_INIT or use other annotations. SIP has the same issue here. You don’t auto-generate from the headers, you modify them to include your own directives and annotations and create a complete specification file.

• cwrap - I’m on a Mac so I used this version for clang. https://github.com/geggo/cwrap Really poor doc - but using the source I finally got it to run and it generated…. an empty .pyx file from a pretty simple header of structs. Not so good.

• xdress - This showed promise. The website is down so the docs are actually seemingly here. There’s an impressive amount of work gone into it and it looks straightforward to use. But it needed all the llvm headers (and a correctly linked version of clang). I had to use brew install llvm —with-clang. There is a xdressclang-3.5 branch, but it doesn’t seem to have enough fixes done. I tried tapping homebrew/versions for an earlier version of clang (install llvm33 / llvm34) and that got it built. Anyway, I digress… it worked great for a simple example, but the resulting ctypes files for the full library was pretty garbled and refused to build. Something in the AST C->Python is a bit awry...

• ctypesgen wasn’t one I had encountered in the original search. The documentation is pretty sparse - or you might call it concise. It hasn’t seemingly had much work done on it the last 4 years either (and people enquiring on the issues list if the developers are ever going to further the project). I’ve tried running it, but sadly it seems to fall over with what I suspect/seems like issues with the Clang compiler cdefs.h use of _attribute_. I’ve tried things like -std=c11 but to no avail.

In conclusion, out of all the ones I’ve looked at I think xdress came the closest to the fully automated generation of python bindings. It worked fine for the simple examples given, but couldn’t handle the more complex existing library headers, with all the complexities of forward declarations, enumerated types, void pointers… It seems a well designed and (for a while) well maintained project, so there is possibly some way to circumvent these issues if someone were to take it on again.

Still, the question remains, does anyone have a robust toolchain for generating python wrappers from C headers automatically? I think the reality is there always has to be a bit of manual work, and for that CFFI looks the most “modern” approach (one of the best overviews/comparisons I encountered is here) - yet it always involves a specially edited cdef() version of any header files (e.g. Using Python's CFFI and excluding system headers).

Answer 1

I find ctypesgen great for autogeneration. I'm only using it with one or two python modules that I hope to open source, and I've been happy so far. Here's a quick example using it with zlib, but I also just tried it successfully with a few other libraries:

(Edit: I know you mentioned ctypesgen has problems on a mac, so maybe it needs someone to tweak it to work on OSX - I don't have OSX at home or I'd try it.)

Get ctypesgen:

git clone https://github.com/davidjamesca/ctypesgen.git

Run short script to call ctypesgen (replace zlib info with another library):

import os                           

ZLIB_INC_DIR = "/usr/include"       
ZLIB_LIB_DIR = "/usr/lib/x86_64-linux-gnu"       
ZLIB_LIB = "libz.so"       
ZLIB_HEADERS = "/usr/include/zlib.h"       

# Set location of ctypesgen.py                                      
ctypesgen_path = 'ctypesgen/ctypesgen.py'                                       

wrapper_filename = 'zlib.py'                                                 
cmd = "LD_LIBRARY_PATH={} {} -I {} -L {} -l {} {} -o {}".format(             
ZLIB_LIB_DIR, ctypesgen_path, ZLIB_INC_DIR, ZLIB_LIB_DIR, ZLIB_LIB,      
ZLIB_HEADERS, wrapper_filename)                                        

print(cmd)
os.system(cmd)                                                             

Usage example:

python
>>> import zlib
>>> zlib.compress("asdfasdfasdfasdfasdf")
'x\x9cK,NIKD\xc3\x00T\xfb\x08\x17'