Broken pipe error with multiprocessing.Queue

Question

In python2.7, multiprocessing.Queue throws a broken error when initialized from inside a function. I am providing a minimal example that reproduces the problem.

#!/usr/bin/python
# -*- coding: utf-8 -*-

import multiprocessing

def main():
    q = multiprocessing.Queue()
    for i in range(10):
        q.put(i)

if __name__ == "__main__":
    main()

throws the below broken pipe error

Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
IOError: [Errno 32] Broken pipe

Process finished with exit code 0

I am unable to decipher why. It would certainly be strange that we cannot populate Queue objects from inside a function.

Answer 1

When You fire up Queue.put(), implicit thread is started to deliver data to a queue. Meanwhile, main application is finished and there is no ending station for the data (queue object is garbage-collected).

I would try this:

from multiprocessing import Queue

def main():
    q = Queue()
    for i in range(10):
        print i
        q.put(i)
    q.close()
    q.join_thread()

if __name__ == "__main__":
    main()

join_thread() ensures, all data in the buffer has been flushed. close() must be called before join_thread()