ending the life of a thread in python?

Question

I have the below code but it lives on after the queue is empty, any insights:

def processor():
   while(1>0):
    if queue.empty() == True:  
    print "the Queue is empty!"  
    break
   source=queue.get()
   page = urllib2.urlopen(source)
   print page   

def main:
   for i in range(threads):  
    th = Thread(target=processor)  
    th.setDaemon(True)  
    th.start()  
   queue.join()  

It prints queue empty as many times as I have threads and just stands there doing nothing.

Answer 1

You need to call queue.task_done() after printing the page, otherwise join() will block. Each thread, after using get() must call task_done().

See documentation for queue

Answer 2

This part:

   while(1>0):
    if queue.empty() == True:  
    print "the Queue is empty!"  
    break

Above is just plain wrong. queue.get() is blocking, there is absolutely no reason to have a busy loop. It should be deleted.

Your code should look something like this.

def processor():
   source=queue.get()
   page = urllib2.urlopen(source)
   print page   
   queue.task_done()

def main:
   for i in range(threads):  
    th = Thread(target=processor)  
    th.setDaemon(True)  
    th.start()  
   for source in all_sources:
    queue.put(source)
   queue.join()  

It's not the cleanest way to exit, but it will work. Since processor threads are set to be daemons, whole process with exit as soon as the main is done.