Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Detect when multiprocessing queue is empty and closed

Let's say I have two processes: a reader and a writer. How does the writer detect when the reader has finished writing values?

The multiprocessing module has a queue with a close method that seems custom-built for this purpose. But how do you detect when the queue has been closed?

This doesn't seem to work, as the writer never exits:

import multiprocessing as mp

def getter(q):
    while True:
        try:
            print(q.get())
        except Exception:
            break

def putter(q):
    q.put(1)
    q.put(2)
    q.close()

q = mp.Queue()
writer = mp.Process(target=putter, args=(q, ))
reader = mp.Process(target=getter, args=(q, ))
reader.start()
writer.start()

writer.join()
reader.join()

Should the reader-writer pair use a sentinel value to signal end of writing? Then what's the point of having the close method?

EDIT: While this question asks about the Queue module (now queue), I am asking specifically about mp.Queue and what the correct use of the .close method is.

like image 368
Daniel Kats Avatar asked Feb 01 '18 18:02

Daniel Kats


People also ask

How do you know if a multiprocessing queue is empty?

The empty() method checks if a multiprocessing queue is empty if the method returns True if the queue is empty. Otherwise, it returns False .

Is multiprocessing queue process safe?

Queues are thread and process safe.

How do you flush a queue in Python?

To clear all items from a queue in Python:queue. clear() . The clear method will remove all elements from the queue.

How do I clear a multiprocessing queue?

There is no direct way of clearing a multiprocessing. Queue . I believe the closest you have is close() , but that simply states that no more data will be pushed to that queue, and will close it when all data has been flushed to the pipe.


1 Answers

But how do you detect when the queue has been closed?

You don't. That is not the purpose of close. Calling close doesn't even guarantee that no more items will be added to the queue; the docs say

Indicate that no more data will be put on this queue by the current process.

close is intended to shut down the current process's feeder thread for that queue (or at least start shutting it down), not to communicate an end-of-queue to other processes.


If you want to signal that no more values will be written to the queue, use standard techniques like enqueueing a sentinel object, like you would with an ordinary queue.Queue.

like image 141
user2357112 supports Monica Avatar answered Oct 17 '22 19:10

user2357112 supports Monica