What is the most efficient way to copy an externally provided buffer to bytes

Question

I am interfacing to an external library using ctypes. This library returns to me a binary buffer. The interface looks like this:

int getBuff(unsigned char **buf, int *len);

The library also exports a deallocator so that I can free the buffer when I am done with it, but that aspect presents no problems to me, so I don't think we need to cover it.

In my ctypes code I am representing the buf argument as c_void_p. I would like to copy this buffer into a bytes object as efficiently as possible.

At the moment I have:

data = bytes(bytearray(ctypes.cast(buf, ctypes.POINTER(ctypes.c_ubyte*len.value))[0]))

where buf is c_void_p and len is c_int.

As I understand it, this performs two copies. Once to the bytearray object, and then again to the bytes object.

How can I do this with only a single copy?

My current efforts have concentrated on Python 2, but in due course I will need to support this for Python 3 as well.

Answer 1

Apparently you can slice a ctypes pointer. Not c_void_p, c_char_p, or c_wchar_p, but POINTER types work. For a POINTER(c_char), slicing it gives you bytes:

data = ctypes.POINTER(ctypes.c_char).from_buffer(buf)[:len.value]

Thanks to eryksun for bringing that up. Also, it's not clear why buf is a c_void_p instead of already being a POINTER(c_char). (For a POINTER(c_char), the code would be just buf[:len.value].)

---

For getting bytes from a general object that supports the buffer protocol, memoryview(...).tobytes() involves one less copy than bytes(bytearray(...)):

data = memoryview(ctypes.cast(buf, ctypes.POINTER(ctypes.c_ubyte*len.value))[0]).tobytes()

This is compatible with both Python 2 and Python 3.

---

Keep in mind that the buf here needs to be a pointer to the buffer, not a pointer to a pointer to the buffer. getBuff takes a pointer to a pointer (so probably byref(buf)).