Using the buffer API in Cython

Question

I'm working in with a C library that repeatedly calls a user supplied function pointer to get more data. I'd like to write a Cython wrapper in such a way that the Python implementation of that callback can return any reasonable data type like str, bytearray, memory mapped files, and so on (specifically, supports the Buffer interface). what I have so far is:

from cpython.buffer cimport PyBUF_SIMPLE
from cpython.buffer cimport Py_buffer
from cpython.buffer cimport PyObject_GetBuffer
from cpython.buffer cimport PyBuffer_Release
from libc.string cimport memmove

cdef class _callback:
    cdef public object callback
    cdef public object data

cdef uint16_t GetDataCallback(void * userdata,
                              uint32_t wantlen, unsigned char * data,
                              uint32_t * gotlen):

    cdef Py_buffer gotdata
    box = <_callback> userdata
    gotdata_object = box.callback(box.data, wantlen)
    if not PyObject_CheckBuffer(gotdata_object):
        # sulk
        return 1

    try:
        PyObject_GetBuffer(gotdata_object, &gotdata, PyBUF_SIMPLE)

        if not (0 < gotdata.len <= wantlen):
            # sulk
            return 1

        memmove(data, gotdata.buf, gotdata.len)

        return 0
    finally:
        PyBuffer_Release(&gotdata)

The code I want to write would produce equivalent C code, but look like this:

from somewhere cimport something
from libc.string cimport memmove

cdef class _callback:
    cdef public object callback
    cdef public object data

cdef uint16_t GetDataCallback(void * userdata,
                              uint32_t wantlen, unsigned char * data,
                              uint32_t * gotlen):


    cdef something gotdata
    box = <_callback> userdata
    gotdata = box.callback(box.data, wantlen)
    if not (0 < gotdata.len <= wantlen):
        # sulk
        return 1

    memmove(data, gotdata.buf, gotdata.len)

    return 0

The generated C code looks like what I think it should be doing; but this seems like digging around in the Python API unnecessarily. Does Cython provide a nicer syntax to achieve this effect?

Answer 1

If you want to support everything that implements every variation of the new-style or old-style buffer interface, then you have to use the C API.

But if you don't care about old-style buffers, you can almost always use a memoryview:

Cython memoryviews support nearly all objects exporting the interface of Python new style buffers. This is the buffer interface described in PEP 3118. NumPy arrays support this interface, as do Cython arrays. The “nearly all” is because the Python buffer interface allows the elements in the data array to themselves be pointers; Cython memoryviews do not yet support this.

This of course includes str (or, in 3.x, bytes), bytearray, etc—if you followed the link, you may notice that it links to the same page to explain what it supports that you linked to explain what you want to support.

For 1D arrays of characters (like str), it's:

cdef char [:] gotdata