multiprocessing.Queue and Queue.Queue are different?

Question

If I use Queue.Queue, then my read() function doesn't work, why? But if I use multiprocessing.Queue, it works well:

from multiprocessing import Pool, Process, Queue 
import os, time
# from Queue import Queue 

def write(q): 
    for v in ['A', 'B', 'C']: 
        print 'Put %s to queue ' % v 
        q.put_nowait(v) 
        time.sleep(0.2) 

def read(q): 
    while 1: 
        if not q.empty(): 
            v = q.get(True) 
            print "Get %s from queue" % v 
            time.sleep(0.2) 
        else: 
            break 

if __name__ == '__main__': 
    q = Queue() 
    pw = Process(target=write, args=(q, )) 
    pr = Process(target=read, args=(q, )) 
    pw.start() 
    pw.join() 

    pr.start() 
    pr.join() 

    print "all done..."

Answer 1

Queue.Queue is just an in-memory queue that knows how to deal with multiple threads using it at the same time. It only works if both the producer and the consumer are in the same process.

Once you have them in separate system processes, which is what the multiprocessing library is about, things are a little more complicated, because the processes no longer share the same memory. You need some kind of inter-process communication method to allow the two processes to talk to each other. It can be a shared memory, a pipe or a socket, or possibly something else. This is what multiprocessing.Queue does. It uses pipes to provide a way for two processes to communicate. It just happens to implement the same API as Queue.Queue, because most Python programmers are already familiar with it.

Also note that the way you are using the queue, you have a race condition in your program. Think about what happens if the write process writes to the queue right after you call q.empty() in the read process. Normally you would add some special item to the queue (e.g. None) which would mean that the consumer can stop.