error checking pipe commands made in python for unix

Question

I am compiling a command to be executed on a unix system from python, containing multiple piping step. e.g.:

grep foo myfile | cmd1 | cmd2 > output

These correspond to a series of transformations on entries that have foo in them from myfile. I sometimes construct it as a command to be executed by os.system and in other cases using the subprocess module.

How can I do error checking on each part of the pipe, from Python? For example in 99% of cases, there are foo entries in myfile and the remaining pipe works. But if for some reason myfile exists but contains no foo entries, then the rest of the pipe breaks because you are piping empty files, and the remaining commands require non-empty inputs to work. This makes it hard to debug because all you see is the output of a broken pipe, not knowing which intermediate step failed.

Is there a way to construct pipes in Python that error check intermediate steps? E.g. check that the grep portion of the pipe really contains some output, that the cmd1 output on grep's input also contained some output, and so on? thanks.

Answer 1

Here is my approach, tested on Python 2.6 Linux.

a.Define a method

def run_command(command):
    p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    out, err = p.communicate()
    return (p.returncode, out, err)

b.Use it in the following way

def test_run_command_pipestatus(self):
    # the exit $PIPESTATUS is the return code of "exit 2". The default return code is the return code of last command in the pipe
    return_code, out, err = run_command("exit 2 | tail -n 1; exit $PIPESTATUS")
    print "return_code = %d" %return_code
    print out
    print err
    self.assertEqual(2, return_code)