How to wrap a c++ function which takes in a function pointer in python using SWIG

Question

Here is a simplified example of what I want to do. Suppose I have the following c++ code in test.h

double f(double x);
double myfun(double (*f)(double x));

It doesn't really matter for now what these functions do. The important thing is that myfun takes in a function pointer.

After including the test.h file in my interface file, I compiled a python module "test" using SWIG. Now, in Python, I run the following commands:

import test
f = test.f

This creates a properly working function f, which takes in a double. However, when I try to pass "f" into myfun within python this is what happens:

myfun(f)
TypeError: in method 'myfun', argument 1 of type 'double (*)(double)'

How do I fix this? I figure I need a typemap declaration in my SWIG interface file, but I am not sure what the proper syntax is or where to put it. I tried

%typemap double f(double);

but that didn't work. Any ideas?

Answer 1

Note: this answer has a long section on workarounds. If you simply want to use this skip straight to solution 5.

The problem

You've run into the fact that in Python everything is an object. Before we look at fixing things, first let's understand what's going on as is. I've created a complete example to work with, with a header file:

double f(double x) {
  return x*x;
}

double myfun(double (*f)(double x)) {
  fprintf(stdout, "%g\n", f(2.0));
  return -1.0;
}

typedef double (*fptr_t)(double);
fptr_t make_fptr() {
  return f;
}

The main changes I've made so far are adding a definition to your declarations so I can test them and a make_fptr() function that returns something to Python we know is going to be wrapped as a function pointer.

With this the first SWIG module might look like:

%module test

%{
#include "test.h"
%}

%include "test.h"

And we can compile it with:

swig2.0 -Wall -python test.i && gcc -Wall -Wextra -I/usr/include/python2.6 -std=gnu99 -shared -o _test.so test_wrap.c

So now we can run this and ask Python about the types we have - the type of test.f and the type of the result of calling test.make_fptr()):

Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> type(test.f)
<type 'builtin_function_or_method'>
>>> repr(test.f)
'<built-in function f>'
>>> type(test.make_fptr())
<type 'SwigPyObject'>
>>> repr(test.make_fptr())
"<Swig Object of type 'fptr_t' at 0xf7428530>"

So the problem as it stands should become clear - there's no conversion from built-in functions to the SWIG type for function pointers, so your call to myfun(test.f) won't work.

The solution

The question then is how (and where) do we fix this? In fact there are at least four possible solutions we might pick, depending on how many other languages you target and how "Pythonic" you want to be.

Solution 1:

The first solution is trivial. We already used test.make_fptr() to return us a Python handle to a function pointer for the funciton f. So we can infact call:

f=test.make_fptr()
test.myfun(f)

Personally I don't like this solution very much, it's not what Python programmers expect and it's not what C programmers expect. The only thing going for it is the simlicity of implementation.

Solution 2:

SWIG gives us a mechanism to expose function pointers to the target language, using %constant. (Normally this is used for exposing compile-time constants, but that's esentially all function pointers really are in their simplest form anyway).

So we can modify our SWIG interface file:

%module test

%{
#include "test.h"
%}

%constant double f(double);
%ignore f;

%include "test.h"

The %constant directive tells SWIG to wrap f as a function pointer, not a function. The %ignore is needed to avoid a warning about seeing multiple versions of the same identifier.

(Note: I also removed the typedef and make_fptr() function from the header file at this point)

Which now lets us run:

Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> type(test.f)
<type 'SwigPyObject'>
>>> repr(test.f)
"<Swig Object of type 'double (*)(double)' at 0xf7397650>"

Great - it's got the function pointer. But there's a snag with this:

>>> test.f(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'SwigPyObject' object is not callable

Now we can't call test.f from the Python side. Which leads to the next solution:

Solution 3:

To fix this let's first expose test.f as both a function pointer and a built-in function. We can do that by simply using %rename instead of %ignore:

%module test

%{
#include "test.h"
%}

%constant double f(double);
%rename(f_call) f;

%include "test.h"

Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> repr(test.f)
"<Swig Object of type 'double (*)(double)' at 0xf73de650>"
>>> repr(test.f_call)
'<built-in function f_call>'

That's a step, but I still don't like the idea of having to remember if I should write test.f_call or just test.f depending on the context of what I want to do with f at the time. We can achieve this just by writing some Python code in our SWIG interface:

%module test

%{
#include "test.h"
%}

%rename(_f_ptr) f;
%constant double f(double);
%rename(_f_call) f;

%feature("pythonprepend") myfun %{
  args = f.modify(args)
%}

%include "test.h"

%pythoncode %{
class f_wrapper(object):
  def __init__(self, fcall, fptr):
    self.fptr = fptr
    self.fcall = fcall
  def __call__(self,*args):
    return self.fcall(*args)
  def modify(self, t):
    return tuple([x.fptr if isinstance(x,self.__class__) else x for x in t])

f = f_wrapper(_f_call, _f_ptr)
%}

There are several functional bits here. Firstly we create a new, pure Python class to wrap a function as both a callable and a function pointer. It holds as members the real SWIG wrapped (and renamed) function pointer and function. These are now renamed to begin with an underscore as Python convention. Secondly we set test.f to be an instance of this wrapper. When it is called as a function it passes the call through. Finally we insert some extra code into the myfun wrapper to swap in the real function pointer rather than our wrapper, taking care not to alter any other arguments if there were any.

This does work as expected, for example with:

import test
print "As a callable"
test.f(2.0)
print "As a function pointer"
test.myfun(test.f)

We could make this a bit nicer, for example with a SWIG macro to avoid repeating the %rename, %constant and wrapper instance creation, but we can't really escape from the need to use %feature("pythonprepend") everywhere we pass these wrappers back to SWIG. (If it is possible to do that transparently it's well beyond my Python knowledge).

Solution 4:

The previous solution is somewhat neater, it works transparently as you'd expect (as both a C and Python user) and the mechanics of it is encapsulated with nothing but Python implementing it.

There is a gotcha still though, besides the need to use pythonprepend for every single usage of the function pointers - if you run swig -python -builtin it simply won't work, because there's no Python code to prepend in the first place! (You'd need to change the construction of the wrapper to be: f = f_wrapper(_test._f_call, _test._f_ptr), but that won't be sufficient).

So we can work around that by writing some Python C API in our SWIG interface:

%module test

%{
#include "test.h"
%}

%{
static __thread PyObject *callback;
static double dispatcher(double d) {
  PyObject *result = PyObject_CallFunctionObjArgs(callback, PyFloat_FromDouble(d), NULL);
  const double ret = PyFloat_AsDouble(result);
  Py_DECREF(result);

  return ret;
}
%}

%typemap(in) double(*)(double) {
  if (!PyCallable_Check($input)) SWIG_fail;
  $1 = dispatcher;
  callback = $input;
}

%include "test.h"

This is a little ugly for two reasons. Firstly it uses a (thread local) global variable to store the Python callable. That's trivially fixable for most real world callbacks, where there's a void* user data argument as well as the actual inputs to the callback. The "userdata" can be the Python callable in those cases.

The second issue is a little more tricky to solve though - because the callable is a wrapped C function the call sequence now involves wrapping everything up as Python types and a trip up and back from the Python interpreter just to do something that should be trivial. That's quite a bit of overhead.

We can work backwards from a given PyObject and try to figure out which function (if any) it is a wrapper for:

%module test

%{
#include "test.h"
%}

%{
static __thread PyObject *callback;
static double dispatcher(double d) {
  PyObject *result = PyObject_CallFunctionObjArgs(callback, PyFloat_FromDouble(d), NULL);
  const double ret = PyFloat_AsDouble(result);
  Py_DECREF(result);

  return ret;
}

SWIGINTERN PyObject *_wrap_f(PyObject *self, PyObject *args);

double (*lookup_method(PyObject *m))(double) {
  if (!PyCFunction_Check(m)) return NULL;
  PyCFunctionObject *mo = (PyCFunctionObject*)m;
  if (mo->m_ml->ml_meth == _wrap_f)
    return f;
  return NULL;
}
%}

%typemap(in) double(*)(double) {
  if (!PyCallable_Check($input)) SWIG_fail;
  $1 = lookup_method($input);
  if (!$1) {
    $1 = dispatcher;
    callback = $input;
  }
}

%include "test.h"

This does need some per function pointer code, but now it's an optimisation rather than a requirement and it could be made more generic right a SWIG macro or two.

Solution 5:

I was working on a neater 5th solution that would use %typemap(constcode) to allow a %constant to be used as both a method and a function pointer. It turns out though that there's already support in SWIG for doing exatly that, which I found when reading some of the SWIG source. So actually all we need to do is simply:

%module test

%{
#include "test.h"
%}

%pythoncallback;
double f(double);
%nopythoncallback;

%ignore f;
%include "test.h"

The %pythoncallback enables some global state that causes subsequent functions to be wrapped as to be usable a both a function pointer and a function! %nopythoncallback disables that.

Which then works (with or without -builtin) with:

import test
test.f(2.0)
test.myfun(test.f)

Which solves almost all the problems in one go. This is even documented in the manual too, although there doesn't seem to be any mention of %pythoncallback. So the previous four solutions are mostly just useful as examples of customising SWIG interfaces.

There is still one case where solution 4 would be useful though - if you want to mix and match C and Python implemented callbacks you would need to implement a hybrid of these two. (Ideally you'd try and do the SWIG function pointer type conversion in your typemap and then iff that failed fallback to the PyCallable method instead).