Sharing a result queue among several processes

Question

The documentation for the multiprocessing module shows how to pass a queue to a process started with multiprocessing.Process. But how can I share a queue with asynchronous worker processes started with apply_async? I don't need dynamic joining or anything else, just a way for the workers to (repeatedly) report their results back to base.

import multiprocessing def worker(name, que):     que.put("%d is done" % name)  if __name__ == '__main__':     pool = multiprocessing.Pool(processes=3)     q = multiprocessing.Queue()     workers = pool.apply_async(worker, (33, q)) 

This fails with: RuntimeError: Queue objects should only be shared between processes through inheritance. I understand what this means, and I understand the advice to inherit rather than require pickling/unpickling (and all the special Windows restrictions). But how do I pass the queue in a way that works? I can't find an example, and I've tried several alternatives that failed in various ways. Help please?

Answer 1

Try using multiprocessing.Manager to manage your queue and to also make it accessible to different workers.

import multiprocessing def worker(name, que):     que.put("%d is done" % name)  if __name__ == '__main__':     pool = multiprocessing.Pool(processes=3)     m = multiprocessing.Manager()     q = m.Queue()     workers = pool.apply_async(worker, (33, q))