Python multiprocessing pipe recv() doc unclear or did I miss anything?

Question

I have been learning how to use the Python multiprocessing module recently, and reading the official doc. In 16.6.1.2. Exchanging objects between processes there is a simple example about using pipe to exchange data.

And, in 16.6.2.4. Connection Objects, there is this statement, quoted "Raises EOFError if there is nothing left to receive and the other end was closed."

So, I revised the example as shown below. IMHO this should trigger an EOFError exception: nothing sent and the sending end is closed.

The revised code:

from multiprocessing import Process, Pipe

def f(conn):
    #conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    #print parent_conn.recv()   # prints "[42, None, 'hello']"
    try:
        print parent_conn.recv()
    except EOFError:
        pass
    p.join()

But, when I tried the revised example on my Ubuntu 11.04 machine, Python 2.7.2, the script hang.

If anyone can point out to me what I missed, I would be very appreciative.

Answer 1

When you start a new process with mp.Process, the child process inherits the pipes of the parent. When the child closes conn, the parent process still has child_conn open, so the reference count for the pipe file descriptor is still greater than 0, and so EOFError is not raised.

To get the EOFError, close the end of the pipe in both the parent and child processes:

import multiprocessing as mp

def foo_pipe(conn):
    conn.close()

def pipe():
    conn = mp.Pipe()
    parent_conn, child_conn = conn
    proc = mp.Process(target = foo_pipe, args = (child_conn, ))
    proc.start()
    child_conn.close()  # <-- Close the child_conn end in the main process too.
    try:
        print(parent_conn.recv())
    except EOFError as err:
        print('Got here')
    proc.join()

if __name__=='__main__':
    pipe()