Sharing a large (read-only) binary string between Python processes? [duplicate]

Question

I have a large, read-only bytes object that I need to operate against across several different Python (3) processes, with each one "returning" (adding to a result queue) a list of results based on their work.

Since this object is very large and read-only, I'd like to avoid copying it into the address space of each worker process. The research I've done suggests that shared memory is the right way to go about this, but I couldn't find a good resource/example of how exactly to do this with the multiprocessing module.

Thanks in advance.

Answer 1

You can use a multiprocessing.Array, which is like ctypes.Array but for shared memory, when given a ctypes type.

# No lock needed, as no write will be done.
array = multiprocessing.Array(ctypes.c_char, long_byte_string, lock=False)

For example:

>>> import multiprocessing
>>> import ctypes
>>> array = multiprocessing.Array(ctypes.c_char, b'\x01\x02\xff\xfe', lock=False)
>>> array[0]
b'\x01'
>>> array[2:]
b'\xff\xfe'
>>> array[:]
b'\x01\x02\xff\xfe'
>>> b'\xff' in array
True