Gevent threads don't finish even though all the Queue items are exhausted

Question

I'm trying to set up a simple producer-consumer system in Gevent but my script doesn't exit:

import gevent
from gevent.queue import *
import time
import random

q = Queue()
workers = []

def do_work(wid, value):
    """
    Actual blocking function
    """
    gevent.sleep(random.randint(0,2))
    print 'Task', value, 'done', wid
    return


def worker(wid):
    """
    Consumer
    """
    while True:
        item = q.get()
        do_work(wid, item)


def producer():
    """
    Producer
    """
    for i in range(4):
        workers.append(gevent.spawn(worker, random.randint(1, 100000)))


    for item in range(1, 9):
         q.put(item)

producer()
gevent.joinall(workers)

I haven't been able to find good examples/tutorials on using Gevent so what I've pasted above is what I've cobbled up from the internet.

Multiple workers get activated, the items go into the queue but even when everything in the queue finishes, the main program doesn't exit. I have to press CTRL ^ C.

What am I doing wrong?

Thanks.

On a side note: if there is anything my script that could be improved, please let me know. Simple things like checking when the Queue is empty, etc.

Answer 1

I think you should use JoinableQueue like in example from documentation.

import gevent
from gevent.queue import *
import time
import random

q = JoinableQueue()
workers = []

def do_work(wid, value):
    gevent.sleep(random.randint(0,2))
    print 'Task', value, 'done', wid

def worker(wid):
    while True:
        item = q.get()
        try:
            do_work(wid, item)
        finally:
            q.task_done()


def producer():
    for i in range(4):
        workers.append(gevent.spawn(worker, random.randint(1, 100000)))

    for item in range(1, 9):
         q.put(item)

producer()
q.join()