IOError Input/Output Error When Printing

Question

I have inherited some code which is periodically (randomly) failing due to an Input/Output error being raised during a call to print. I am trying to determine the cause of the exception being raised (or at least, better understand it) and how to handle it correctly.

When executing the following line of Python (in a 2.6.6 interpreter, running on CentOS 5.5):

print >> sys.stderr, 'Unable to do something: %s' % command

The exception is raised (traceback omitted):

IOError: [Errno 5] Input/output error

For context, this is generally what the larger function is trying to do at the time:

from subprocess import Popen, PIPE
import sys
def run_commands(commands):
    for command in commands:
        try:
            out, err = Popen(command, shell=True, stdout=PIPE, stderr=PIPE).communicate()
            print >> sys.stdout, out
            if err:
                raise Exception('ERROR -- an error occurred when executing this command: %s --- err: %s' % (command, err))
        except:
            print >> sys.stderr, 'Unable to do something: %s' % command
run_commands(["ls", "echo foo"])

The >> syntax is not particularly familiar to me, it's not something I use often, and I understand that it is perhaps the least preferred way of writing to stderr. However I don't believe the alternatives would fix the underlying problem.

From the documentation I have read, IOError 5 is often misused, and somewhat loosely defined, with different operating systems using it to cover different problems. The best I can see in my case is that the python process is no longer attached to the terminal/pty.

As best I can tell nothing is disconnecting the process from the stdout/stderr streams - the terminal is still open for example, and everything 'appears' to be fine. Could it be caused by the child process terminating in an unclean fashion? What else might be a cause of this problem - or what other steps could I introduce to debug it further?

In terms of handling the exception, I can obviously catch it, but I'm assuming this means I wont be able to print to stdout/stderr for the remainder of execution? Can I reattach to these streams somehow - perhaps by resetting sys.stdout to sys.__stdout__ etc? In this case not being able to write to stdout/stderr is not considered fatal but if it is an indication of something starting to go wrong I'd rather bail early.

I guess ultimately I'm at a bit of a loss as to where to start debugging this one...

Answer 1

I think it has to do with the terminal the process is attached to. I got this error when I run a python process in the background and closed the terminal in which I started it:

$ myprogram.py
Ctrl-Z
$ bg
$ exit

The problem was that I started a not daemonized process in a remote server and logged out (closing the terminal session). A solution was to start a screen/tmux session on the remote server and start the process within this session. Then detaching the session+log out keeps the terminal associated with the process. This works at least in the *nix world.

Answer 2

I had a very similar problem. I had a program that was launching several other programs using the subprocess module. Those subprocesses would then print output to the terminal. What I found was that when I closed the main program, it did not terminate the subprocesses automatically (as I had assumed), rather they kept running. So if I terminated both the main program and then the terminal it had been launched from*, the subprocesses no longer had a terminal attached to their stdout, and would throw an IOError. Hope this helps you.

*NB: it must be done in this order. If you just kill the terminal, (for some reason) that would kill both the main program and the subprocesses.