Understanding os.fork and Queue.Queue

Question

I wanted to implement a simple python program using parallel execution. It's I/O bound, so I figured threads would be appropriate (as opposed to processes). After reading the documentation for Queue and fork, I thought something like the following might work.

q = Queue.Queue()

if os.fork():            # child
    while True:
        print q.get()
else:                    # parent
    [q.put(x) for x in range(10)]

However, the get() call never returns. I thought it would return once the other thread executes a put() call. Using the threading module, things behave more like I expected:

q = Queue.Queue()

def consume(q):
    while True:
        print q.get()

worker = threading.Thread (target=consume, args=(q,))
worker.start()

[q.put(x) for x in range(10)]

I just don't understand why the fork approach doesn't do the same thing. What am I missing?

Answer 1

The POSIX fork system call creates a new process, rather than a new thread inside the same adress space:

The fork() function shall create a new process. The new process (child process) shall be an exact copy of the calling process (parent process) except as detailed below: [...]

So the Queue is duplicated in your first example, rather than shared between the parent and child.

You can use multiprocessing.Queue instead or just use threads like in your second example :)

By the way, using list comprehensions just for side effects isn't good practice for several reasons. You should use a for loop instead:

for x in range(10): q.put(x)

Answer 2

To share the data between unrelated processes, you can use named pipes. Through the os.open() funcion.. http://docs.python.org/2/library/os.html#os.open. You can simply name a pipe as named_pipe='my_pipe' and in a different python programs use os.open(named_pipe, ), where mode is WRONLY and so on. After that you'll make a FIFO to write into the pipe. Don't forget to close the pipe and catch exceptions..