Cython - implementing callbacks

Question

I have been working with Cython in an attempt to interface with a library written in c++. So far things are going pretty good, and I can effectively use MOST functions within the library. My only problem lies within implementing callbacks. The library has 4 function definitions that look a little something like this:

typedef void (*Function1)(const uint16_t *data, 
                         unsigned width, unsigned height);
void SetCallBack(Function1);

So to implement them I figured I would do something like this with cython:

ctypedef void (*Function1)(unsigned short *data, 
                         unsigned width, unsigned height);
cdef extern from "lib.hpp":
    void SetCallBack(Function1)

Which actually compiles correctly, however, I can't for the life of me think how to actually implement that in such a way that the callback would work. I first tried creating a function that would just call that, similar to how you would do it for any other function, coming up with this:

def PySetCallBack(Func):
    SetCallBack(Func)

but that gives me the (predictable) error:

"Cannot convert Python object to 'Function1'"

so yeah, that's where I'm at. If anyone has any experience setting up callbacks in Cython I would be very grateful for any assistance. Thanks.

Edit: Following your advice, I created an intermediate function with a cdef, which looks like this:

cdef void cSetCallBack(Function1 function):
    SetCallBack(function)

This seems to have gotten me... Closer? Getting a different error now at least:

error: invalid conversion from ‘void (*)(short unsigned int*, unsigned int, unsigned int)’ to ‘void (*)(const uint16_t*, unsigned int, unsigned int)’

Now, far as I can tell those types are identical, so I can't figure what's going on.

Edit2: Fixed that problem by declaring a new typedef:

ctypedef unsigned short uint16_t

and using that as the argument to call, but apparently that wasn't actually getting any closer, but just taking me around a side track, since when trying to call that function, I get the same "Cannot convert Python object to 'Function1'" error all over again.

So, I'm pretty much back where I started. Only thing I can figure to do now is explicitly cast the python object coming in as a c function like that, but, to be honest, I have no idea how I would go about that.

Edit the third: Alright, after dissecting your answer I finally get it, and it works, so hooray and whatnot. What I ended up doing was creating a function like this:

cdef void cSetCallback(Function1 function):
    SetCallback(function)
cdef void callcallback(const_ushort *data, unsigned width, unsigned height):
    global callbackfunc
    callbackfunc(data,width,height)
cSetCallback(callcallback)
def PySetCallback(callbackFunc):
    global callbackfunc
    callbackfunc = callbackFunc

So now the only problem is that it can't convert const_ushort *data into a python object, but that's another problem entirely, so I guess this one is solved, thanks a lot.

Answer 1

I've recently been in the situation where I also had to interface an existing C++ library with Python using Cython, making an intensive use of events/callbacks. It was not that easy to find sources about this and I would like to put all of this together here :

First of all, the wrapping C++ callback class (based on 'double (METHOD)(void)' prototype, but it could have been templatized, since Cython can handle templates) :

ALabCallBack.h :

#ifndef ALABCALLBACK_H_
#define ALABCALLBACK_H_


#include <iostream>

using namespace std;

namespace elps {

//template < typename ReturnType, typename Parameter >
class ALabCallBack {
public:

    typedef double (*Method)(void *param, void *user_data);

    ALabCallBack();
    ALabCallBack(Method method, void *user_data);
    virtual ~ALabCallBack();

    double cy_execute(void *parameter);

    bool IsCythonCall()
    {
        return is_cy_call;
    }

protected:

    bool is_cy_call;

private:

    //void *_param;
    Method _method;
    void *_user_data;

};


} /* namespace elps */
#endif /* ALABCALLBACK_H_ */

ALabCallBack.cpp :

#include "ALabCallBack.h"

namespace elps {


ALabCallBack::ALabCallBack() {
    is_cy_call = true;
};

ALabCallBack::~ALabCallBack() {
};

ALabCallBack::ALabCallBack(Method method, void *user_data) {
    is_cy_call = true;
    _method = method;
    _user_data = user_data;
};

double ALabCallBack::cy_execute(void *parameter)
{
    return _method(parameter, _user_data);
};


} /* namespace elps */

Where :

• 'callback' :: The pattern/converter method to fire a Python (=Method) object method from C typed infos

• 'method' :: The effective method passed by the Python user (=user_data)

• 'parameter' :: The parameter to be passed to the 'method'

Now, we need to implement the .pyx file...

Our base prototype :

ctypedef double (*Method)(void *param, void *user_data)

Then, we provide a Cython wrapper for the C++ class :

cdef extern from "../inc/ALabCallBack.h" namespace "elps" :
    cdef cppclass ALabCallBack:
        ALabCallBack(Method method, void *user_data)
        double cy_execute(void *parameter)

The pattern/converter method to be used for translating C typed prototype to a Python object call :

cdef double callback(void *parameter, void *method):
    return (<object>method)(<object>parameter)

Now let's embed this features in a Cython class :

cdef class PyLabCallBack:
    cdef ALabCallBack* thisptr

    def __cinit__(self, method):
        # 'callback' :: The pattern/converter method to fire a Python 
        #               object method from C typed infos
        # 'method'   :: The effective method passed by the Python user 
       self.thisptr = new ALabCallBack(callback, <void*>method)

    def __dealloc__(self):
       if self.thisptr:
           del self.thisptr

    cpdef double execute(self, parameter):
        # 'parameter' :: The parameter to be passed to the 'method'
        return self.thisptr.cy_execute(<void*>parameter)

Edit : Better typing for execute function : def execute => cpdef double

That's it. Call it like doing something like that :

def func(obj):
    print obj 
    obj.Test()     # Call to a specific method from class 'PyLabNode'
    return obj.d_prop

n = PyLabNode()    # Custom class of my own
cb = PyLabCallBack(func)
print cb.execute(n)

As python is implicitly typed, we can access the properties of the 'obj' object related to the class of the object passed as argument when the time comes to fire the callback.

It can be quite easily adapted for pure C implementation. Please, tell me if you can see any possible enhancement for this (in my case, perfs are very crucial since events are fired intensively).

Answer 2

If you can modify the library to define:

typedef void (*Function1)(const uint16_t *data, 
                          unsigned width, unsigned height,
                          void *user_data);
void SetCallBack(Function1, void*);

instead, I fear you are out of luck. If you have the void*, than you define a function that calls a python callable object with correct arguments and SetCallBack with this function and the python callable.

If you can't, but the callback is global (it seems to be), you can create a global variable to store the python object in. Than you'd again create a function to call the python object and pass it to SetCallBack and your PySetCallback would just set the global and ensure proper function is registered.

If the callback is context-specific, but you have no way to pass it a "user data" pointer, I fear you are out of luck here.

I know python and C++, but not cython, so I don't know whether you can create the function in cython, or whether you'd have to write in C++.