Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to iterate through a Python Queue.Queue with a for loop instead of a while loop?

Tags:

python

Normally we code it like this:

while True:
    job = queue.get()
    ...

But is it also possible to do something in the lines of:

for job in queue.get():
    #do stuff to job

The real reason why I want to do that is because I wanted to use python-progressbar's auto detect maxval. They do it like for this in progressbar(that):

like image 912
majidarif Avatar asked Jan 16 '14 09:01

majidarif


People also ask

Can you iterate through a queue Python?

Use a while loop to iterate through a queue in Python, e.g. while not q. empty(): . The loop checks if the queue is not empty and iterates as long as there are items in the queue.

How do you use a while loop instead of a for loop in Python?

The for statement iterates through a collection or iterable object or generator function. The while statement simply loops until a condition is False. It isn't preference. It's a question of what your data structures are.

Can you iterate through a queue?

Using std::queue In other words, std::queue is not meant to be iterated over. If you need to iterate over a std::queue , you can create a copy of it and remove items from the copy, one at a time, using the standard pop function after processing it.

Can you iterate through a stack in Python?

You can also iterate over it with for item in stack if you want FIFO or for item in reversed(stack) for FILO, which will generate a memory efficient reverse iterator.


4 Answers

You can use iter with callable. (You should pass two arguments, one for the callable, the other for the sentinel value)

for job in iter(queue.get, None): # Replace `None` as you need.     # do stuff with job 

NOTE This will block when no elements remain and no sentinel value is put. Also, like a while-get loop and unlike normal for loops over containers, it will remove items from the queue.

None is common value, so here's a sample with more concrete sentinel value:

sentinel = object() for job in iter(queue.get, sentinel):     # do stuff with job 
like image 159
falsetru Avatar answered Sep 30 '22 19:09

falsetru


I would say this is an easy way to iterate over queue in some points:

from queue import Queue  q = Queue() q.put(1) q.put(2) q.put(3)  for i in q.queue:     print(i) 
like image 29
shakil18 Avatar answered Sep 30 '22 21:09

shakil18


For that kind of queue actually I would not typically use this check of queue.empty() because I always use it in a threaded context and thus cannot know whether another thread would put something in there in a few milliseconds (thus that check would be useless anyway). I never check a queue for being empty. I rather use a sentinel value which marks the ending of a producer.

So using the iter(queue.get, Sentinel) is more what I like.

If you know that no other thread will put items in the queue anymore and just want to drain it from all currently contained items, then you can use sth like this:

class Drainer(object):
  def __init__(self, q):
    self.q = q
  def __iter__(self):
    while True:
      try:
        yield self.q.get_nowait()
      except queue.Empty:  # on python 2 use Queue.Empty
        break

for item in Drainer(q):
  print(item)

or

def drain(q):
  while True:
    try:
      yield q.get_nowait()
    except queue.Empty:  # on python 2 use Queue.Empty
      break

for item in drain(q):
  print(item)
like image 34
Alfe Avatar answered Sep 30 '22 19:09

Alfe


My first though was for the iter function, but the built in queue module doesn't return a sentinel, so a good alternative might be to define your own wrapper class:

import Queue

class IterableQueue():
    def __init__(self,source_queue):
            self.source_queue = source_queue
    def __iter__(self):
        while True:
            try:
               yield self.source_queue.get_nowait()
            except Queue.Empty:
               return

This iterator wraps the queue and yields until the queue is empty, then returns, so now you can do:

q = Queue.Queue()
q.put(1)
q.put(2)
q.put(3)

for n in IterableQueue(q):
    print(n)

Output:

1
2
3

This method is a bit verbose it would be interesting if anyone knows anything better using the builtins.

like image 33
James Avatar answered Sep 30 '22 21:09

James