Python fork(): passing data from child to parent

Question

I have a main Python process, and a bunch or workers created by the main process using os.fork().

I need to pass large and fairly involved data structures from the workers back to the main process. What existing libraries would you recommend for that?

The data structures are a mix of lists, dictionaries, numpy arrays, custom classes (which I can tweak) and multi-layer combinations of the above.

Disk I/O should be avoided. If I could also avoid creating copies of the data -- for example by having some kind of shared-memory solution -- that would be nice too, but is not a hard constraint.

For the purposes of this question, it is mandatory that the workers are created using os.fork(), or a wrapper thereof that would clone the master process's address space.

This only needs to work on Linux.

Answer 1

multiprocessing's queue implementation works. Internally, it pickles data to a pipe.

q = multiprocessing.Queue()
if (os.fork() == 0):
    print(q.get())
else:
    q.put(5)
# outputs: 5