c malloc array pointer return in cython

Question

How does one to return a malloc array pointer (or numpy array pointer) in cython back to python3, efficiently.

The cython code works perfectly as long as I don't return the array pointer

I would like:

def double complex* randn_zig(int n):
  ...
  r = malloc(n*n*sizeof(double complex))
  ...
  return r

The c11 (gcc 11) equivalent is:

double complex* randn_zig(int n){

    r = malloc(n*n*sizeof(double complex))

    return r
}

I have tried <double complex*> randn_zig(int n):

and randn_zig(<double complex*> r, int n):

and other permutations without success so far. The c and cython code version is 5 times as fast as Numby/ pylab randn version if I can find a way to return a pointer to the large 10^6 to 10^10 double complex array.

Answer 1

Numpy C API

Your question is similar to this post.

You can use the function below to pass a C pointer to Numpy array. The memory will be freed automatically when the Numpy array is recycled. If you want free the pointer mamully, you should not set NPY_OWNDATA flag.

import numpy as np
cimport numpy as np

cdef pointer_to_numpy_array_complex128(void * ptr, np.npy_intp size):
    '''Convert c pointer to numpy array.
    The memory will be freed as soon as the ndarray is deallocated.
    '''
    cdef extern from "numpy/arrayobject.h":
        void PyArray_ENABLEFLAGS(np.ndarray arr, int flags)
    cdef np.ndarray[np.complex128, ndim=1] arr = \
            np.PyArray_SimpleNewFromData(1, &size, np.NPY_COMPLEX128, ptr)
    PyArray_ENABLEFLAGS(arr, np.NPY_OWNDATA)
    return arr

For reference:

• PyArray_SimpleNewFromData
• Numpy Data Type API

Cython Typed Memoryviews

Of couse, you can also use cython memoryview.

import numpy as np
cimport numpy as np

cdef np.complex128_t[:,:] view = <np.complex128_t[:n,:n]> c_pointer
numpy_arr = np.asarray(view)

The code above will transfer C pointer to a numpy array. However this would not free memory automaticlly, you have to free the memory by yourself or it would lead to memory leak!

Answer 2

A further option (in addition to the two options from the top answer: PyArray_SimpleNewFromData and just returning the typed memoryview without handling the memory) is to use the cython.view.array class.

This is a fairly low-level class that can be used to wrap existing memory. It has an attribute callback_free_data where you can set a function to be called on destruction so that it does free the memory (example code here is copied from the documentation):

cdef view.array my_array = view.array(..., mode="fortran", allocate_buffer=False)
my_array.data = <char *> my_data_pointer

# define a function that can deallocate the data (if needed)
my_array.callback_free_data = free

It exposes the buffer protocol so that you can index it, use it with typed memoryviews, or wrap it with a Numpy array (without copying) with np.asarray. The latter feature may be easier to use than PyArray_SimpleNewFromData.