Cannot open pipe descriptors created by threads in Python

Question

I just began to study the pipe method of python. I tried to wrap the pipe descriptors into file object and read by line.

import os,time,threading

def child():
    while True:
        time.sleep(1)
        msg = ('Spam\n' ).encode()
        os.write(pipeout,msg)


def parent(): 
    while True:
        a = os.fdopen(pipein)
        line = a.readline()[:-1]
        print('Parent %d got [%s] at %s' % (os.getpid(),line,time.time()))

pipein,pipeout = os.pipe()

threading.Thread(target=child,args=()).start()

parent()

when I run the script, the results are following----the script just works in the first iteration and then shows the error messages

Parent 621 got [Spam] at 1376785841.4  
Traceback (most recent call last):
  File "/Users/miteji/pipe-thread.py", line 43, in <module>
    parent()
  File "/Users/miteji/pipe-thread.py", line 36, in parent
    line = a.readline()[:-1]
IOError: [Errno 9] Bad file descriptor
>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py",         line 551, in __bootstrap_inner
     self.run()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 504, in run
     self.__target(*self.__args, **self.__kwargs)
  File "/Users/miteji/pipe-thread.py", line 30, in child
     os.write(pipeout,msg)
OSError: [Errno 32] Broken pipe

However, when I changed

a = os.fdopen(pipein)
line = a.readline()[:-1]

to

line = os.read(pipein,32)

the scrips works fine.

So Why the "os.fdopen" method cannot be used? why the pipe is broken? Thank you all!

Answer 1

The problem lies in the placement of os.fdopen here:

def parent(): 
    while True:
        a = os.fdopen(pipein)
        line = a.readline()[:-1]
        print('Parent %d got [%s] at %s' % (os.getpid(),line,time.time()))

Each trip through the loop, you call os.fdopen() again, even if you did before.

The first time you do it, you did not do any earlier os.fdopen(), so all is well. But the second time, this re-binds a to the new result, abandoning the earlier os.fdopen() value.

When the earlier value is abandoned, it becomes eligible for garbage collection. CPython notices immediately (due to reference counting) and collects it. This deletes the underlying object, which calls os.fdclose(). That, in turn, closes the pipe.

To fix the immediate problem, then, make sure you only open the pipe once, outside the loop.