Cython - converting pointers to arrays into Python objects

Question

Alright, I am so close to finishing this I can taste it. Over the past few week or so, I've been attempting to create a Python extension to interface with a library written in C++ via Cython. With a little help from the guys here and a couple of friends, I have managed to get what feels like 98% of the way there. Only thing remaining is this: I can't for the life of me figure out how to turn a pointer to an array of unsigned shorts into a python object (Preferably a list).

A little background, I am trying to interface with a part of the library that sets a callback function, which I have successfully done with this:

global callbackfunc

ctypedef unsigned short const_ushort "const uint16_t"

ctypedef void (*Function1)(const_ushort *data, unsigned width, unsigned height)

cdef extern from "lib.hpp":
    void SetCallback(Function1)

cdef void cSetCallback(Function1 function):
    SetCallback(function)

cdef void callcallback(const_ushort *data, unsigned width, unsigned height):
    global callbackfunc
    callbackfunc(data,width,height)

cSetCallback(callcallback)

def PySetCallback(callbackFunc):
    global callbackfunc
    callbackfunc = callbackFunc

The problem occurs within the function "callcallback", where I get the error: "Cannot convert 'const_ushort *' to Python object". My first attempt around this was to create a new python list, and loop through to get each element of the array into a python list, like this:

datalist = []
for i in range(width*height):
    datalist += data[i]

Which, sadly, nets me with the compiled cython code trying to define a type as a "const const unsigned short", which is obviously a problem.

Then I tried this:

datalist = []
for i in data:
    datalist += i

Which gives me "C array iteration requires known end index". Note that I know very little C/C++, so most of this doesn't make much sense to me.

So, anyways, is there any effective way of translating a pointer like that into a python object (Preferably faster than looping through the array, since it's usually about 57344 items, and this is quite time sensitive)

Edit: A little more clarification, as I mentioned, I'm working with callbacks, and the C++ function within the library that calls this sends a pointer to an array of "const uint_16"s, which is why I defined const_ushort that way, because otherwise the types don't unify. I cannot modify the library in any way.

Edit2: Looks like I got it. What I ended up having to do was explicitly cast the array as an array of unsigned shorts rather than an array of const unsigned shorts to I could index them with a non constant. To achieve this, I created another C++ function like this (Someone else wrote it for me, I barely know C++):

unsigned short *convert_short(const unsigned short *test){ return const_cast<unsigned short *>(test); }

and that allowed me to create the "getindex" function within my class and return the correct values based on the function. So yeah, Python seems to be reading the arrays correctly and whatnot, so this case seems closed. Thanks a lot.

Answer 1

ctypedef unsigned short const_ushort "const uint16_t"

makes the const appear in the C code, so you get typedef unsigned short const uint16_t in which the const is meaningless. (It also redefines the standard type uint16_t, which it shouldn't.)

You may have more success with

ctypedef unsigned short *ushort_p "ushort_p"

For some reason, Cython seems not to recognize the C const keyword. You may have to use a wrapper around your callbacks to deal with the const issue.

If you want the pointer translated into a Python object, wrap it in a cdef class. Make sure the class records the height and width, since these are not recorded with the C array.

Answer 2

This is not quite what you're asking, but if what you want to do is work with an array of unsigned shorts in Cython and then make your answer accessible on the Python side, you can create a Numpy ndarray on the Python side and pass it into Cython where it is roughly the same as a multi-dimensional array. That way, you get both Python's memory management, some convenient indexing syntax, and the speed of working with a real array in Cython.

Here's an example from my Project Euler solutions.