Menu
I'm writing a wrapper class for use with a workflow manager. I would like to log output from an application (child process executed via subprocess.Popen) in a certain way:
stdout of the child should go to a log file and to stdout of the parent,stderr of the child should go to a different logfile, but also to stdout of the parent.I.e. all output from the child should end up merged on stdout (like with subprocess.Popen(..., stderr=subprocess.STDOUT), so I can reserve stderr for log messages from the wrapper itself. On the other hand, the child's streams should go to different files to allow separate validation.
I've tried using a "Tee" helper class to tie two streams (stdout and the log file) together, so that Tee.write writes to both streams. However, this cannot be passed to Popen because "subprocess" uses OS-level functions for writing (see here: http://bugs.python.org/issue1631).
The problem with my current solution (code snippet below, adapted mostly from here) is that output on stdout may not appear in the right order.
How can I overcome this? Or should I use an altogether different approach?
(If I stick with the code below, how do I choose a value for the number of bytes in os.read?)
import subprocess, select, sys, os
call = ... # set this
process = subprocess.Popen(call, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
logs = {process.stdout: open("out.log", "w"), process.stderr: open("err.log", "w")}
done = {process.stdout: False, process.stderr: False}
while (process.poll() is None) or (not all(done.values())):
ready = select.select([process.stdout, process.stderr], [], [])[0]
for stream in ready:
data = os.read(stream.fileno(), 1)
if data:
sys.stdout.write(data)
logs[stream].write(data)
else:
done[stream] = True
logs[process.stdout].close()
logs[process.stderr].close()
By the way, this solution using "fcntl" has not worked for me. And I couldn't quite figure out how to adapt this solution to my case yet, so I haven't tried it.
972
To capture the output of the subprocess. run method, use an additional argument named “capture_output=True”. You can individually access stdout and stderr values by using “output. stdout” and “output.
If successful, popen() returns a pointer to an open stream that can be used to read or write to a pipe. If unsuccessful, popen() returns a NULL pointer and sets errno to one of the following values: Error Code. Description.
The subprocess module defines one class, Popen and a few wrapper functions that use that class. The constructor for Popen takes arguments to set up the new process so the parent can communicate with it via pipes. It provides all of the functionality of the other modules and functions it replaces, and more.
stdout=PIPE means that subprocess' stdout is redirected to a pipe that you should read e.g., using process.communicate() to read all at once or using process.stdout object to read via a file/iterator interfaces.
If you set shell=True, you can pass a command string to subprocess that includes pipes, redirections, and the tee command.
102
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
