queue.empty and doing a put while empty

Question

Let's say I have a queue of two elements. I loop though the queue popping off the items using get. I'm afraid the loop will stop looping once I pop off the second element, and I need to reprocess it because of some error. So I put it back in the queue, but it won't since by then the queue is empty.

My loop:

while not queue.empty():
    try:
        item = queue.get()
        do stuff(item):
    except Exception as e:
        queue.put(item)
    queue.task_done()

I read the documentation on Queue.empty() but getting confused about blocking and what it means.

Answer 1

If you are working with multiple threads as it looks like to me (why else would you be concerned about blocking?), it is in general not a good idea to do this:

if not queue.empty():  # or 'while' instead of 'if'
    item = queue.get()

because another thread could have emptied your queue between the first and the second line, so your queue.get() now blocks until anything new comes in, which seems not to be expected behavior if you wrote that. Basically the if not queue.empty() is meaningless in a concurrency kind of scenario.

The right way to do this would rather be:

try:
    while True:
        item = queue.get(block=False)
        done = False
        while not done:
            try:
                do stuff(item)
                done = True
            except Exception as e:
                pass  # just try again to do stuff
        queue.task_done()
except Empty:
    pass  # no more items

Other than that (if we assume, no other thread can consume your queue), the code would work fine. I have some serious doubts though about whether you understand (or 'understood', since the question is old) the concept of blocking. If a function is 'blocking' according to the documentation, that just means that the execution of your code will be halted at that point until some event occurs. time.sleep(1) is maybe the best example of a blocking function that blocks for 1 second.

PS: I know this is some years old, but still...