Python waiting for a queue and an event

Question

I have a queue and an event. I would like to exit the loop when the event is set to True, however there is a queue.get() in the loop which blocks until there is something in it.

How can I abort the waiting of the self._commandQueue.get() when the closeEvent Event flag is set?

Note: I want to avoid depending on the blocking nature of the queue and want to block based on the condition of the queue and the eventflag

def _execute(self):
    while not self._closeEvent.isSet():
        nextCommand = self._commandQueue.get()
        self._commandExecutor.execute(nextCommand)
        self._commandQueue.task_done()

Answer 1

You would need something like the Windows WaitForMultipleObjects() call, but the python event and queue API don't offer such a beast (but you could use win32api to use that if you are strictly windows), so if you really need BOTH event sources to be checked in parallel, the answer is 'you cannot without polling (or monkey patching the Event class to allow it)'.

But if you are a bit more flexible, you can arrange something like it, by redefining your command queue a bit. If the command queue is a PriorityQueue, you could queue your normal jobs with normal priority and have an extra process queue a 'STOP' token with higher priority, once your event signals.

STOP = None

def _execute(self):
    while 1:
        nextCommand = self._commandQueue.get()[1]
        if nextCommand is STOP:
           break
        self._commandExecutor.execute(nextCommand)
        self._commandQueue.task_done()

def wait_for_stop_signal(self):
    self._closeEvent.wait()
    self._commandQueue.put((-1, STOP))

Now you run wait_for_stop_signal in its own thread, and you have the behaviour you want (but waste one thread instead of polling, pick whats worse for your use case).

Answer 2

Queue uses events internally according to http://hg.python.org/cpython/file/3a1db0d2747e/Lib/Queue.py#l150 Specifically it uses self.not_empty event which your application could also wait on before attempting a Queue.get_nowait call.

As for waiting on several events there's Python threading: can I sleep on two threading.Event()s simultaneously? question with some code examples.

Answer 3

Queue.get is a blocking method when it's called without parameters.

From docs:

Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Full exception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (timeout is ignored in that case).

You need to do something like that:

try:
    # If `False`, the program is not blocked, it will throw the Queue.Empty exception.
    my_data = queue.get(False)  
    .....Some Code......
except Queue.Empty:
    my_data = None # or whatever you want

More options

1. get_nowait:

   Queue.get_nowait()

   which is equivalent to get(False).

2. Using timeout:

   my_data = queue.get(True, 5)

   This will try to get for 5 secends if the get will fail (nothing to get) it will raise the same exception Queue.Empty