Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python threads and queue example

I'm new to python (I come from PHP), I've been reading tutorials and trying things for a couple of days but I can't understand this queue example (http://docs.python.org/2/library/queue.html)

def worker():
    while True:
        item = q.get()
        do_work(item)
        q.task_done()

q = Queue()
for i in range(num_worker_threads):
     t = Thread(target=worker)
     t.daemon = True
     t.start()

for item in source():
    q.put(item)

q.join()       # block until all tasks are done

The thing I don't understand is how the worker thread completes and exists. I've read q.get() blocks until an item is available, so if all items are processed and none is left in the queue why q.get() doesn't block forever?

like image 483
nacholibre Avatar asked Mar 02 '13 11:03

nacholibre


People also ask

How do you queue and thread in Python?

You can make a queue or line of tasks or objects by using the queue library in Python. Simply you can add a task to the queue (using put() method) or get a task out of the line and processes it (using get() method). Threading package in Python let you run multiple tasks at the same time.

Are queues in Python thread-safe?

Yes, Queue is thread-safe.

What is thread in Python with example?

What Is a Thread? A thread is a separate flow of execution. This means that your program will have two things happening at once. But for most Python 3 implementations the different threads do not actually execute at the same time: they merely appear to.

What is queue in Python example?

Like stack, queue is a linear data structure that stores items in First In First Out (FIFO) manner. With a queue the least recently added item is removed first. A good example of queue is any queue of consumers for a resource where the consumer that came first is served first.


1 Answers

Threads do not exit normally in this code (they are indeed blocked when the queue is empty). The program doesn't wait for them because they're daemon threads.

The program doesn't exit immediately and doesn't block forever because of q.join and q.task_done calls.

The count of unfinished tasks goes up whenever an item is added to the queue. The count goes down whenever a consumer thread calls task_done() to indicate that the item was retrieved and all work on it is complete. When the count of unfinished tasks drops to zero, join() unblocks, and the program exists without waiting for daemon threads.

like image 114
Pavel Anossov Avatar answered Sep 20 '22 16:09

Pavel Anossov