Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Good way to build a blocking, zero-length queue in Python

In Java, there's java.util.concurrent.SynchronousQueue, a queue with no storage capacity. Threads trying to put/get value always block until another thread tries to get/put a value respectively.

What are good ways to do the same in Python? I.e. I want a way to pass values from a set of one or more threads to another set of one or more threads without a value ever "belonging" to a thread in either group.

Python's queue.Queue does not allow the length to be 0, specifying a non-positive value for the maximum capacity creates an unbounded queue.

like image 212
Feuermurmel Avatar asked Apr 30 '14 14:04

Feuermurmel


People also ask

What Python data type is best for a queue?

deque can be a good choice for queue data structure in Python's standard library.

Is Python queue get blocking?

get() method in python is a blocking function.

What is the max size of queue in Python?

To answer your 1st question: for all intents and purposes, the max size of a Queue is infinite. The reason why is that if you try to put something in a Queue that is full, it will wait until a slot has opened up before it puts the next item into the queue.


2 Answers

You can use Queue.join() and Queue.task_done() to block until the get() has completed:

class SynchronousQueue(object):

    def __init__(self):
        self.q = Queue(1)
        self.put_lock = RLock()

    def get(self):
        value = self.q.get(block=True)
        self.q.task_done()
        return value

    def put(self, item):
        with self.put_lock:
            self.q.put(item, block=True)
            self.q.join()
like image 69
Dan Getz Avatar answered Sep 18 '22 15:09

Dan Getz


I have the feeling the following might be deadlock city, but would something like the following work?

class SynchronousQueue(object):
    def __init__(self):
        self.ready_to_get = Queue(1)
        self.queue = Queue(1)

    def get(self):
        self.ready_to_get.put('ready', block=True)
        return self.queue.get(block=True)

    def put(self, item):
        self.ready_to_get.get(block=True)
        self.queue.put(item, block=True)

A regular queue supports half of what you want (the getter waiting on the putter), so we can try and implement the reverse by blocking the put until a get has started.

like image 28
Jon Betts Avatar answered Sep 20 '22 15:09

Jon Betts