Python multiprocessing Pipe with large object will hang

Question

I have a simple code snippet below that demonstrates the problem.

from multiprocessing import Pipe
import time

recv_end, send_end = Pipe(duplex=False)
d = {'word'+str(elem): elem for elem in range(3000)}

start_time = time.time()
send_end.send(d)
print('--- %s seconds ---' % (time.time()-start_time))

The above works fine and is fast enough for my purposes, no problem there. But if I make the size to 5000, it simply hangs indefinitely:

from multiprocessing import Pipe
import time

recv_end, send_end = Pipe(duplex=False)
d = {'word'+str(elem): elem for elem in range(5000)}  # changed to 5000

start_time = time.time()
send_end.send(d)
print('--- %s seconds ---' % (time.time()-start_time))

Is there a size limit for Pipe, or is this a non reproducible problem? How about if you make the size even bigger? And if there is a size limit, what is the best way to avoid this problem and send over the large dictionary through Pipe? Thanks in advance!

Answer 1

This problem occurs is that Pipe.send() is a blocking call and it waits to be received. Read more here. To make it work, you can create process like in following code:

#!/usr/bin/env python
from multiprocessing import Pipe, Process
import time
import sys


def foo(conn):
    d = {'word'+str(elem): elem for elem in range(5000)}  # changed to 5000
    conn.send(d)
    conn.close()


recv_end, send_end = Pipe(duplex=False)
p = Process(target=foo, args=(send_end, ))
p.start()

start_time = time.time()
recv_end.recv()  # Try to comment and you will see that it waits for being received
p.join()
print('--- %s seconds ---' % (time.time()-start_time))