Merging a Python script's subprocess' stdout and stderr while keeping them distinguishable

Question

I would like to direct a python script's subprocess' stdout and stdin into the same file. What I don't know is how to make the lines from the two sources distinguishable? (For example prefix the lines from stderr with an exclamation mark.)

In my particular case there is no need for live monitoring of the subprocess, the executing Python script can wait for the end of its execution.

Answer 1

tsk = subprocess.Popen(args,stdout=subprocess.PIPE,stderr=subprocess.STDOUT) 

subprocess.STDOUT is a special flag that tells subprocess to route all stderr output to stdout, thus combining your two streams.

btw, select doesn't have a poll() in windows. subprocess only uses the file handle number, and doesn't call your file output object's write method.

to capture the output, do something like:

logfile = open(logfilename, 'w')  while tsk.poll() is None:     line = tsk.stdout.readline()     logfile.write(line) 

Answer 2

I found myself having to tackle this problem recently, and it took a while to get something I felt worked correctly in most cases, so here it is! (It also has the nice side effect of processing the output via a python logger, which I've noticed is another common question here on Stackoverflow).

Here is the code:

import sys import logging import subprocess from threading import Thread  logging.basicConfig(stream=sys.stdout,level=logging.INFO) logging.addLevelName(logging.INFO+2,'STDERR') logging.addLevelName(logging.INFO+1,'STDOUT') logger = logging.getLogger('root')  pobj = subprocess.Popen(['python','-c','print 42;bargle'],      stdout=subprocess.PIPE, stderr=subprocess.PIPE)  def logstream(stream,loggercb):     while True:         out = stream.readline()         if out:             loggercb(out.rstrip())                else:             break  stdout_thread = Thread(target=logstream,     args=(pobj.stdout,lambda s: logger.log(logging.INFO+1,s)))  stderr_thread = Thread(target=logstream,     args=(pobj.stderr,lambda s: logger.log(logging.INFO+2,s)))  stdout_thread.start() stderr_thread.start()  while stdout_thread.isAlive() and stderr_thread.isAlive():      pass 

Here is the output:

STDOUT:root:42 STDERR:root:Traceback (most recent call last): STDERR:root:  File "<string>", line 1, in <module> STDERR:root:NameError: name 'bargle' is not defined 

You can replace the subprocess call to do whatever you want, I just chose running python with a command that I knew would print to both stdout and stderr. The key bit is reading stderr and stdout each in a separate thread. Otherwise you may be blocking on reading one while there is data ready to be read on the other.