Extending python - to swig, not to swig or Cython

Question

I found the bottleneck in my python code, played around with psycho etc. Then decided to write a c/c++ extension for performance.

With the help of swig you almost don't need to care about arguments etc. Everything works fine.

Now my question: swig creates a quite large py-file which does a lot of 'checkings' and 'PySwigObject' before calling the actual .pyd or .so code.

Does anyone of you have any experience whether there is some more performance to gain if you hand-write this file or let swig do it.

Answer 1

You should consider Boost.Python if you are not planning to generate bindings for other languages as well with swig.

If you have a lot of functions and classes to bind, Py++ is a great tool that automatically generates the needed code to make the bindings.

Pybindgen may also be an option, but it's a new project and less complete that Boost.Python.

---

Edit:

Maybe I need to be more explicit about pro and cons.

• Swig:

  pro: you can generate bindings for many scripting languages.

  cons: I don't like the way the parser works. I don't know if the made some progress but two years ago the C++ parser was quite limited. Most of the time I had to copy/past my .h headers add some % characters and give extra hints to the swig parser.

  I was also needed to deal with the Python C-API from time to time for (not so) complicated type conversions.

  I'm not using it anymore.

• Boost.Python:

  pro: It's a very complete library. It allows you to do almost everything that is possible with the C-API, but in C++. I never had to write C-API code with this library. I also never encountered bug due to the library. Code for bindings either works like a charm or refuse compile.

  It's probably one of the best solutions currently available if you already have some C++ library to bind. But if you only have a small C function to rewrite, I would probably try with Cython.

  cons: if you don't have a pre-compiled Boost.Python library you're going to use Bjam (sort of make replacement). I really hate Bjam and its syntax.

  Python libraries created with B.P tend to become obese. It also takes a lot of time to compile them.

• Py++ (discontinued): it's Boost.Python made easy. Py++ uses a C++ parser to read your code and then generates Boost.Python code automatically. You also have a great support from its author (no it's not me ;-) ).

  cons: only the problems due to Boost.Python itself. Update: As of 2014 this project now looks discontinued.

• Pybindgen:

  It generates the code dealing with the C-API. You can either describe functions and classes in a Python file, or let Pybindgen read your headers and generate bindings automatically (for this it uses pygccxml, a python library wrote by the author of Py++).

  cons: it's a young project, with a smaller team than Boost.Python. There are still some limitations: you cannot use multiple inheritance for your C++ classes, Callbacks (not automatically, custom callback handling code can be written, though). Translation of Python exceptions to C.

  It's definitely worth a good look.

• A new one: On 2009/01/20 the author of Py++ announced a new package for interfacing C/C++ code with python. It is based on ctypes. I didn't try it already but I will! Note: this project looks discontiued, as Py++.

• CFFI: I did not know the existence of this one until very recently so for now I cannot give my opinion. It looks like you can define C functions in Python strings and call them directly from the same Python module.

• Cython: This is the method I'm currently using in my projects. Basically you write code in special .pyx files. Those files are compiled (translated) into C code which in turn are compiled to CPython modules. Cython code can look like regular Python (and in fact pure Python are valid .pyx Cython files), but you can also more information like variable types. This optional typing allows Cython to generate faster C code. Code in Cython files can call both pure Python functions but also C and C++ functions (and also C++ methods).

  It took me some time to think in Cython, that in the same code call C and C++ function, mix Python and C variables, and so on. But it's a very powerful language, with an active (in 2014) and friendly community.