Python multiprocessing: RuntimeError: "Queue objects should only be shared between processes through inheritance"

Question

I am aware of multiprocessing.Manager() and how it can be used to create shared objects. In particular, queues which can be shared among workers. There is this question, this question, and this question.

However, these links don't mention why we can use inheritance for sharing between processes. As I understand, a queue can still only be copied in this case.

Answer 1

The Queue implementation in python relies on a system pipe to transmit the data from one process to another and some semaphores to protect the read and write on this pipe.

The pipe is handled as an open file in the process and can only be shared with a subprocess at spawning time, because of OS constraints.
The semaphores are also treated as files that should only be shared at spawning time, at least in UNIX based system, for early version of python.

As these 2 sub objects cannot be shared in general, the Queue cannot be pickled and sent to a subprocess once it has been started.

However, for some OS and recent version of python, it is possible to share the Connection and to create sharable Semaphore. Thus, you could in theory create your own Queue that can be shared between processes. But it involves a lot of hacks and might not be very secured.