Capture the error output of a foreground command using plumbum

Question

I'm using the plumbum python library (http://plumbum.readthedocs.org/) as a replacement for shell scripts.

There's a command I want to run that, when it fails it outputs the paths to a file I'm interested in:

$ slow_cmd
Working.... 0%
Working.... 5%
Working... 15%
FAIL. Check log/output.log for details

I want to run the program on the foreground to check the progress:

from plumbum.cmd import slow_cmd

try:
    f = slow_cmd & FG
except Exception, e:
    print "Something went wrong."

# Need the error output from f to get the log file :(    

When the slow_cmd fails, it throws the exception (which I can catch). But I cannot obtain the error output from the exception or from the f future object.

If I don't run the slow_cmd on the FG, the exception contains all of the output and I can read the file from there.

Answer 1

the problem is, FG redirects the output straight to your program's stdout. see https://github.com/tomerfiliba/plumbum/blob/master/plumbum/commands.py#L611

when output is redirected this way, it doesn't go through plumbum's machinery so you won't get it in the exception object. if you're willing to block until slow_cmd finishes, a better solution would be to read from stdout yourself. here's a sketch:

lines = []
p = slow_cmd.popen()
while p.poll() is None:
    line = p.stdout.readline()
    lines.append(line)
    print line
if p.returncode != 0:
    print "see log file..."

a more elegant solution would be to write your own ExecutionModifier (like FG) that duplicates the output streams. let's call it TEE (after http://en.wikipedia.org/wiki/Tee_(command))... i haven't tested it, but it should do the trick (minus selecting on stdout/err):

class TEE(ExecutionModifier):
    def __init__(self, retcode = 0, dupstream = sys.stdout):
        ExecutionModifier.__init__(self, retcode)
        self.dupstream = dupstream
    def __rand__(self, cmd):
        p = cmd.popen()
        stdout = []
        stderr = []
        while p.poll():
            # note: you should probably select() on the two pipes, or make the pipes nonblocking,
            # otherwise readline would block
            so = p.stdout.readline()
            se = p.stderr.readline()
            if so:
                stdout.append(so)
                dupstream.write(so)
            if se:
                stderr.append(se)
                dupstream.write(se)
        stdout = "".join(stdout)
        stderr = "".join(stderr)
        if p.returncode != self.retcode:
            raise ProcessExecutionError(p.argv, p.returncode, stdout, stderr)
        return stdout, stderr

try:
    stdout, stderr = slow_cmd & TEE()
except ProcessExecutionError as e:
    pass # find the log file, etc.