ctx parameter in multiprocessing.Queue

Question

I´m trying to use a Queue from the multiprocessing.Queue module. The Implementation (https://docs.python.org/3.4/library/multiprocessing.html#exchanging-objects-between-processes) uses

q = Queue()

as an example for instantiation. If i try this, i get the following error:

TypeError: __init__() missing 1 required keyword-only argument: 'ctx'

Googling the problem brought up this:

http://bugs.python.org/issue21367

How do i know if this is fixed? Is it impossible to use multiprocessing.Queues right now? If not, how do i get the needed ctx object (and what is it?)

Answer 1

It sounds like you're not importing Queue directly from multiprocessing. When contexts were introduced, most of the objects you import from the multiprocessing top-level package became functions which internally get a context, then pass that to an underlying class initializer, rather than being classes themselves. For example, here is what multiprocessing.Queue is now:

def Queue(self, maxsize=0):
    '''Returns a queue object'''
    from .queues import Queue
    return Queue(maxsize, ctx=self.get_context())

If you were to import multiprocessing.queues.Queue directly and try to instantiate it, you'll get the error you're seeing. But it should work fine if you import it from multiprocessing directly.

The context object tells multiprocessing which of the available methods for starting sub-processes is in use. multiprocessing.Queue uses a multiprocessing.Lock internally, which has to know the correct context to function properly.

Answer 2

This is how should multiprocessing Queue class be inherited from Python 3.4 and on:

from multiprocessing.queues import Queue

class BlockedQueue(Queue):
    def __init__(self, maxsize=-1, block=True, timeout=None):
        self.block = block
        self.timeout = timeout
        super().__init__(maxsize, ctx=multiprocessing.get_context())