Python objects as userdata in ctypes callback functions

Question

The C function myfunc operates on a larger chunk of data. The results are returned in chunks to a callback function:

int myfunc(const char *data, int (*callback)(char *result, void *userdata), void *userdata);

Using ctypes, it's no big deal to call myfunc from Python code, and to have the results being returned to a Python callback function. This callback work fine.

myfunc = mylib.myfunc
myfunc.restype = c_int
myfuncFUNCTYPE = CFUNCTYPE(STRING, c_void_p)
myfunc.argtypes = [POINTER(c_char), callbackFUNCTYPE, c_void_p]

def mycb(result, userdata):
    print result
    return True

input="A large chunk of data."
myfunc(input, myfuncFUNCTYPE(mycb), 0)

But, is there any way to give a Python object (say a list) as userdata to the callback function? In order to store away the result chunks, I'd like to do e.g.:

def mycb(result, userdata):
    userdata.append(result)

userdata=[]

But I have no idea how to cast the Python list to a c_void_p, so that it can be used in the call to myfunc.

My current workaround is to implement a linked list as a ctypes structure, which is quite cumbersome.

Answer 1

I guess you could use the Python C API to do that... maybe you could use a PyObject pointer.

edit: As the op pointed out in the comments, there's already a py_object type readily available in ctypes, so the solution is to create first a ctypes.py_object object from the python list and then casting it to c_void_p to pass it as an argument to the C function (I think this step might be unnecessary as a parameter typed as void* should accept any pointer, and it would be faster to pass just a byref). In the callback, the reverse steps are done (casting from the void pointer to a pointer to py_object and then getting the value of the contents).

A workaround could be to use a closure for your callback function so it already knows in which list it has to append the items...

myfunc = mylib.myfunc
myfunc.restype = c_int
myfuncFUNCTYPE = CFUNCTYPE(STRING)
myfunc.argtypes = [POINTER(c_char), callbackFUNCTYPE]


def mycb(result, userdata):
    userdata.append(result)

input="A large chunk of data."
userdata = []
myfunc(input, myfuncFUNCTYPE(lambda x: mycb(x, userdata)))