Python writable buffer/memoryview to array/bytearray/ctypes string buffer

Question

Problem:

• Binary data of fixed size records
• Want to use struct.unpack_from and struct.pack_into to manipulate the binary data
• Want no copies of the data
• Want multiple views into the memory to simply offset calculations etc.
• Data could be in an array.array bytearray or ctypes string buffer

What I tried to do:

part1 = buffer(binary_data, 0, size1)
part2 = buffer(binary_data, size1, size2)
part3 = buffer(binary_data, size1 + size2) # no size is given for this one as it should consume the rest of the buffer
struct.pack_into('I', part3, 4, 42)

The problem here is that struct.pack_into complains about the buffers being read only. I have looked into memoryviews as they can create a read/write view however they don't allow you to specify the offset and size like the buffer function does.

How can I accomplish having multiple zero-copy views into a buffer of bytes that is readable,writable and can be accessed/modified using struct.unpack_from and struct.pack_into

Answer 1

In 2.6+, ctypes data types have a from_buffer method that takes an optional offset. It expects a writable buffer and will raise an exception otherwise. (For readonly buffers there's from_buffer_copy.) Here's a quick translation of your example to use ctypes char arrays:

from ctypes import *
import struct

binary_data = bytearray(24)
size1 = size2 = 4
size3 = len(binary_data) - size1 - size2

part1 = (c_char * size1).from_buffer(binary_data)
part2 = (c_char * size2).from_buffer(binary_data, size1)
part3 = (c_char * size3).from_buffer(binary_data, size1 + size2)
struct.pack_into('4I', part3, 0, 1, 2, 3, 4)

>>> binary_data[8:]
bytearray(b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00')

>>> struct.unpack_from('4I', part3)
(1, 2, 3, 4)