Cython: "Cannot convert Python object" error

Question

I'm trying to get a test project working that calls a C function, with an array parameter, from Python:

test.cpp:

void testFn(int arr[]);

void testFn(int arr[])
{
    arr[0] = 1;
    arr[1] = 2;
} 

caller.pyx:

import ctypes

cdef extern from "test.cpp":
    void testFn(int arr[])

def myTest():
    a = [0, 0]
    arr = a.ctypes.data_as(ctypes.POINTER(ctypes.c_integer))
    testFn(arr)
    print(arr)

setup.caller.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

sourcefiles = ['caller.pyx']
ext_modules = [Extension("caller", sourcefiles)]

setup(
    name = 'test app',
    cmdclass = {'build_ext': build_ext},
    ext_modules = ext_modules
)

But when I try to build the project I get an error:

$ python setup.caller.py build_ext --inplace
running build_ext
cythoning caller.pyx to caller.c

Error compiling Cython file:
------------------------------------------------------------
...
def myTest():
    a = [0, 0]
    arr = a.ctypes.data_as(ctypes.POINTER(ctypes.c_integer))
    testFn(arr)
          ^
------------------------------------------------------------

caller.pyx:13:11: Cannot convert Python object to 'int *'

Answer 1

This question pops up so often, but I was not able to find a good duplicate-target, which would go beyond "just do that and it will work".

This is a very usual situation: you try to pass some python data structures to c-code which expects pointers int *, double *,.... However pointers are not a python object, so we cannot pass them from/to python code.

Cython can automatically handle conversion to int, double, float and so on, even char * (it is a null-terminated c-string) and some stl-containers, but not to pointers (char *being the one exception).

There are two most common situations:

1. the internal representation of the python-data is already a c-array (array.array, numpy-arrays).
2. the internal representation isn't a continuous array (e.g. list)

1. Passing via memory-view:

There is no way in python we can somehow get hold of a pointer, so this must be done in cython. My first choice to pass an array.array or numpy.array (or any other python-object supporting buffer-protocol, for ctypes.Array see for example this SO-question) to a cython-function would be a memory-view:

def myTest(int[:] arr):
    testFn(&arr[0])

and now calling it from python:

>>> import array
>>> a=array.array('i', [0]*2)
>>> import caller
>>> caller.myTest(a)
>>> a
array('i', [1, 2]) #it worked

The following is important

1. int[:] is a python object (a typed memory view) so it can be passed to a python function (def or cpdef).
2. &arr[0] is used to take the address of the buffer of the memory view. Its result is of type int *.
3. The call of the function is "type-safe", you cannot pass for example array.array('I', [0]*2) to it, because it is not a int-memory-view but a unsigned int-memory-view.

2. Passing non-continuous memory (e.g. lists):

There is more work with list and Co.: The information is not stored in plain c-arrays so we need to copy them first to a continuous memory, pass this temp variable to our c-code and copy the results back to list, our cython function could looks as follows

import array
def myTest2(lst):
    tmp=array.array('i', lst)
    myTest(tmp)
    lst.clear()
    lst.extend(tmp)

And now after reloading the caller module:

>>> lst=[0,0]
>>> caller.myTest2(lst)
[1, 2]

So, it is possible to pass the content of a list to a c-function, but basically you want to work with array.array or numpy.array if you need data-exchange with c-code.