How to wrap a C pointer and length in a new-style buffer object in Cython?

Question

I'm writing a Python 2.7 extension module in Cython. How do I create a Python object implementing the new-style buffer interface that wraps a chunk of memory given to me by a C library? The chunk of memory is just a string of bytes, not a structure or multidimensional array. I'm given a const void * pointer and a length, and some details about how long the pointer stays valid.

I can't copy the memory—that would kill performance for my application.

With the old-style buffer objects I could simply use PyBuffer_FromMemory(), but I can't seem to find a similarly easy way to produce a new-style buffer object.

Do I have to create my own class that implements the buffer interface? Or does Cython provide an easy way to do this?

I've read the Unicode and Passing Strings and Typed Memoryviews pages from the Cython documentation, but the documentation is imprecise and not very complete and there are no examples that look similar to what I want to do.

Here's what I've tried (test.pyx):

from libc.stdlib cimport malloc
from libc.string cimport memcpy

## pretend that this function is in some C library and that it does
## something interesting.  (this function is unrelated to the problem
## I'm experiencing -- this is just an example function that returns a
## chunk of memory that I want to wrap in an object that follows the
## new buffer protocol.)
cdef void dummy_function(const void **p, size_t *l):
    cdef void *tmp = malloc(17)
    memcpy(tmp, "some test\0 bytes", 17)
    p[0] = tmp
    l[0] = 17

cpdef getbuf():
    cdef const void *cstr
    cdef size_t l
    dummy_function(&cstr, &l)

    ## error: test.pyx:21:20: Invalid base type for memoryview slice: void
    #cdef const void[:] ret = cstr[:l]

    ## error: test.pyx:24:9: Assignment to const 'ret'
    #cdef const char[:] ret = cstr[:l]

    ## error: test.pyx:27:27: Cannot convert 'void const *' to memoryviewslice
    #cdef char[:] ret = cstr[:l]

    ## this next attempt cythonizes, but raises an exception:
    ## $ python -c 'import test; test.getbuf()'
    ## Traceback (most recent call last):
    ##   File "<string>", line 1, in <module>
    ##   File "test.pyx", line 15, in test.getbuf (test.c:1411)
    ##   File "test.pyx", line 38, in test.getbuf (test.c:1350)
    ##   File "stringsource", line 614, in View.MemoryView.memoryview_cwrapper (test.c:6763)
    ##   File "stringsource", line 321, in View.MemoryView.memoryview.__cinit__ (test.c:3309)
    ## BufferError: Object is not writable.
    cdef char[:] ret = (<const char *>cstr)[:l]

    ## this raises the same exception as above
    #cdef char[:] ret = (<char *>cstr)[:l]

    return ret

Answer 1

You can define an extension type that implements the buffer protocol by defining the __getbuffer__ and __releasebuffer__ special methods. For example:

from cpython.buffer cimport PyBuffer_FillInfo
from libc.stdlib cimport free, malloc
from libc.string cimport memcpy

cdef void dummy_function(const void **p, size_t *l):
    cdef void *tmp = malloc(17)
    memcpy(tmp, "some test\0 bytes", 17)
    p[0] = tmp
    l[0] = 17

cdef void free_dummy_data(const void *p, size_t l, void *arg):
    free(<void *>p)

cpdef getbuf():
    cdef const void *p
    cdef size_t l
    dummy_function(&p, &l)
    return MemBuf_init(p, l, &free_dummy_data, NULL)

ctypedef void dealloc_callback(const void *p, size_t l, void *arg)

cdef class MemBuf:
    cdef const void *p
    cdef size_t l
    cdef dealloc_callback *dealloc_cb_p
    cdef void *dealloc_cb_arg

    def __getbuffer__(self, Py_buffer *view, int flags):
        PyBuffer_FillInfo(view, self, <void *>self.p, self.l, 1, flags)
    def __releasebuffer__(self, Py_buffer *view):
        pass

    def __dealloc__(self):
        if self.dealloc_cb_p != NULL:
            self.dealloc_cb_p(self.p, self.l, self.dealloc_cb_arg)

# Call this instead of constructing a MemBuf directly.  The __cinit__
# and __init__ methods can only take Python objects, so the real
# constructor is here.  See:
# https://mail.python.org/pipermail/cython-devel/2012-June/002734.html
cdef MemBuf MemBuf_init(const void *p, size_t l,
                        dealloc_callback *dealloc_cb_p,
                        void *dealloc_cb_arg):
    cdef MemBuf ret = MemBuf()
    ret.p = p
    ret.l = l
    ret.dealloc_cb_p = dealloc_cb_p
    ret.dealloc_cb_arg = dealloc_cb_arg
    return ret

With the above (named test.pyx) you get the following behavior:

$ python -c 'import test; print repr(memoryview(test.getbuf()).tobytes())'
'some test\x00 bytes\x00'

I don't know if there's an easier way.

Answer 2

Python 3.3 has PyMemoryView_FromMemory C-API function, which creates a memoryview Python object from supplied C buffer. memoryview objects indeed implement new-style buffer interface.

If you look into its sources, you'll notice that they're rather simple. It does that same thing as PyMemoryView_FromBuffer does, except the former fills Py_buffer with PyBuffer_FillInfo itself.

Since the latter one exists in Python 2.7, so why we can't just call PyBuffer_FillInfo ourselves?

from libc.stdlib cimport malloc
from libc.string cimport memcpy

cdef extern from "Python.h":
    ctypedef struct PyObject
    object PyMemoryView_FromBuffer(Py_buffer *view)
    int PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int infoflags)
    enum:
        PyBUF_FULL_RO

cdef void dummy_function(const void **p, size_t *l):
    cdef void *tmp = malloc(17)
    memcpy(tmp, "some test\0 bytes", 17)
    p[0] = tmp
    l[0] = 17

cpdef getbuf():
    cdef const void *cstr
    cdef size_t l
    cdef Py_buffer buf_info
    cdef char[:] ret
    cdef int readonly

    dummy_function(&cstr, &l)

    readonly = 1
    PyBuffer_FillInfo(&buf_info, NULL, <void*>cstr, l, readonly, PyBUF_FULL_RO)
    ret = PyMemoryView_FromBuffer(&buf_info)

    return ret

Note that, however, that the returned value will have a repr that looks like this: <MemoryView of 'memoryview' at 0x7f216fc70ad0>. This is because Cython seems to wrap bare memoryview inside _memoryviewslice. Since memoryview objects implement buffer interface already, you should probably simply return the result of PyMemoryView_FromBuffer call instead.

Additionally, you're responsible for managing the lifetime of your buffer. memoryview objects created this way will not free memory automatically. You must do it yourself, ensuring that you only do that once no memorybuffer references it. In this regard, answer by Richard Hansen is much better alternative.

Answer 3

As @RichardHansen correctly observes in his self-answer, what you want is a class that implements the buffer protocol, and has a suitable destructor that manages the memory.

Cython actually provides a fairly lightweight class built into it in the form of cython.view.array so there's no need to create your own. It's actually documented in the page you linked but for the sake of providing a quick example that fits your case:

# at the top of your file
from cython.view cimport array

# ...

# after the call to dummy_function
my_array = array(shape=(l,), itemsize=sizeof(char), format='b',  # or capital B depending on if it's signed
                 allocate_buffer=False)
my_array.data = cstr
my_array.callback_free_data = free

cdef char[:] ret = my_array

Just to draw attention to a couple of bits: allocate_buffer is set to False since you're allocating your own in cstr. Setting callback_free_data ensures that the standard library free function is used.