Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Force NumPy ndarray to take ownership of its memory in Cython

Following this answer to "Can I force a numpy ndarray to take ownership of its memory?" I attempted to use the Python C API function PyArray_ENABLEFLAGS through Cython's NumPy wrapper and found it is not exposed.

The following attempt to expose it manually (this is just a minimum example reproducing the failure)

from libc.stdlib cimport malloc
import numpy as np
cimport numpy as np

np.import_array()

ctypedef np.int32_t DTYPE_t

cdef extern from "numpy/ndarraytypes.h":
    void PyArray_ENABLEFLAGS(np.PyArrayObject *arr, int flags)

def test():
    cdef int N = 1000

    cdef DTYPE_t *data = <DTYPE_t *>malloc(N * sizeof(DTYPE_t))
    cdef np.ndarray[DTYPE_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &N, np.NPY_INT32, data)
    PyArray_ENABLEFLAGS(arr, np.NPY_ARRAY_OWNDATA)

fails with a compile error:

Error compiling Cython file:
------------------------------------------------------------
...
def test():
    cdef int N = 1000

    cdef DTYPE_t *data = <DTYPE_t *>malloc(N * sizeof(DTYPE_t))
    cdef np.ndarray[DTYPE_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &N, np.NPY_INT32, data)
    PyArray_ENABLEFLAGS(arr, np.NPY_ARRAY_OWNDATA)
                          ^
------------------------------------------------------------

/tmp/test.pyx:19:27: Cannot convert Python object to 'PyArrayObject *'

My question: Is this the right approach to take in this case? If so, what am I doing wrong? If not, how do I force NumPy to take ownership in Cython, without going down to a C extension module?

like image 567
kynan Avatar asked May 26 '14 15:05

kynan


People also ask

Does NumPy work with Cython?

You can use NumPy from Cython exactly the same as in regular Python, but by doing so you are losing potentially high speedups because Cython has support for fast access to NumPy arrays.

Does Cython speed up NumPy?

By explicitly declaring the "ndarray" data type, your array processing can be 1250x faster. This tutorial will show you how to speed up the processing of NumPy arrays using Cython. By explicitly specifying the data types of variables in Python, Cython can give drastic speed increases at runtime.


1 Answers

You just have some minor errors in the interface definition. The following worked for me:

from libc.stdlib cimport malloc
import numpy as np
cimport numpy as np

np.import_array()

ctypedef np.int32_t DTYPE_t

cdef extern from "numpy/arrayobject.h":
    void PyArray_ENABLEFLAGS(np.ndarray arr, int flags)

cdef data_to_numpy_array_with_spec(void * ptr, np.npy_intp N, int t):
    cdef np.ndarray[DTYPE_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &N, t, ptr)
    PyArray_ENABLEFLAGS(arr, np.NPY_OWNDATA)
    return arr

def test():
    N = 1000

    cdef DTYPE_t *data = <DTYPE_t *>malloc(N * sizeof(DTYPE_t))
    arr = data_to_numpy_array_with_spec(data, N, np.NPY_INT32)
    return arr

This is my setup.py file:

from distutils.core import setup, Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("_owndata", ["owndata.pyx"])]
setup(cmdclass={'build_ext': build_ext}, ext_modules=ext_modules)

Build with python setup.py build_ext --inplace. Then verify that the data is actually owned:

import _owndata
arr = _owndata.test()
print arr.flags

Among others, you should see OWNDATA : True.

And yes, this is definitely the right way to deal with this, since numpy.pxd does exactly the same thing to export all the other functions to Cython.

like image 168
Stefan Avatar answered Nov 10 '22 03:11

Stefan