python multiprocessing manager.list inside manager.dict

Question

The following python code:

from multiprocessing import Manager

manager = Manager()
globals = manager.dict()
globals["queue"] = manager.Queue()

#following line fails
globals["queue"].put("Starting")

fails with error:

  File "<string>", line 2, in __getitem__
  File "/usr/lib/python3.5/multiprocessing/managers.py", line 732, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Unserializable message: ('#RETURN', <queue.Queue object at 0x7feb68a68ef0>)
---------------------------------------------------------------------------

Can anyone explain why?

Answer 1

In short, you are abusing the thread-safe dict by attempting to store non-serializable queue (which is thread-safe on its own). The best way to go about it is to use independent variables to store the collections created with manager:

d = manager.dict()
q = manager.Queue()

then q.put("Starting") works. You need to pass it directly to the function or method that is going to be executed in a separate process, e.g.:

def f(d,q):
    d['a'] = 1
    q.put('a')

p = Process(target=f, args=(d,q,))

multiprocessing.manager is supposed to be used as the provider of thread-safe collections that can be reused by threads and processes. The caveat is that objects originating from the manager cannot hold other objects that were created using it.

I recommend a read of the documentation of multiprocessing module which is very friendly and has a lot of good examples to start with.