Cython: create C function that returns an array

Question

I would like to create a Cython function that reads an array and returns an array. This function would be called from within other cdef functions, not python def functions. Here's what I have.

In my .pxd file:

cdef int[:] array_test(double *x) nogil

In my .pyx file:

cdef inline int[:] array_test(double *x) nogil:

    cdef int output[2]
    output[0]=1
    output[1]=9

    return output

But when I compile, I get the error: "Operation not allowed without gil" Can someone please help?

Answer 1

Probably there is a misunderstanding: this function doesn't return a c-array but a memory view slice. You don't have to believe me, you can check it by deleting nogil and calling cython. In the created *.c-file you can see the C-signature of your function, __Pyx_memviewslice being the important part:

 static CYTHON_INLINE __Pyx_memviewslice __pyx_f_4file_array_test(CYTHON_UNUSED double *__pyx_v_x)

This memory view is a Python-object, so it has to update its reference counter (at least when created) and thus the Global Interpreter Lock is needed (otherwise another thread could mess up this very counter and the object will not be destroyed or destroyed while it is still used somewhere) - that is the reason you see the "Operation not allowed without gil"-error message.

So you have at least three options:

1. Is it really so important to do it with "nogil"? If not, then just drop it. It is the simplest solution, the downside: you might lose performance.
2. Use a real C-array, i.e. int *res = (int *) malloc(2*sizeof(int)). It is fast, the downside: you have to manage the memory by yourself.
3. Use c++ and std::vector<int>, the advantage is, that you don't have to manage the memory any longer, but you will need to switch to c++.

---

An improvement of possibility 1. is to aquire gil only for the last line, where it is needed (this will not make much difference for this example but might for real code):

cdef inline int[:] array_test(double *x) nogil:
    cdef int output[2]
    output[0]=1
    output[1]=9
    with gil:
        return output

---

As OP has proposed, a further possibility would be to change the signature of the function to:

cdef inline void array_test(double *x, int[:] output) nogil:

The trick here: the fuction array_test no longer creates the resulting memory view slice and thus doesn't have to do reference counting - which makes "nogil" possible. Btw, it is only possible, because for cdef-functions Cython doesn't call Py_INCREF/Py_DECREF for arguments (which it has to do for def-functions).

There are some minor downsides, like the calling of array_test becoming more cumbersome and the caller must have gil in order to create the output memory view. But it has also the advantage, that the caller can determine in which kind of data structure the result should be stored (numpy array or maybe something else).