Logging from an External Application

Question

I am writing a research tool and I have recently switched from using "print" statements to using the logger functionality built into Python. This, I reasoned, would allow me to give the user the option of dumping the output to a file, besides dumping it to the screen.

So far so good. The part of my code that is in Python uses "logger.info" and "logger.error" to dump to both the screen and a file. "logger" is the module-wide logger. This part works like a charm.

However, at several points, I use "subprocess.call" to run an executable through the shell. So, throughout the code, I have lines like

proc = subprocess.call(command)

The output from this command would print to the screen, as always, but it would not dump to the file that the user specified.

One possible option would be to open up a pipe to the file:

proc = subprocess.call(command, stdout=f, stderr=subprocess.OUTPUT)

But that would only dump to the file and not to the screen.

Basically, my question boils down to this: is there a way I can leverage my existing logger, without having to construct another handler for files specifically for subprocess.call? (Perhaps by redirecting output to the logger?) Or is this impossible, given the current setup? If the latter, how can I improve the setup?

(Oh, also, it would be great if the logging were in 'real time', so that messages from the executable are logged as they are received.)

Thanks for any help! :)

Answer 1

Instead of piping stdout to a file, you can pipe it to a PIPE, and then read from that PIPE and write to logger. Something like this:

proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.OUTPUT)
for line in proc.stdout:
    logging.info(line)

However, there's an even simpler answer: You have to use a file-like object with a file handle, but you can create one on top of pipes that passes each line to logging. You could write this object yourself, but, as @unutbu says, someone's already done it in this question. So:

with StreamLogger(logging.INFO) as out:
    proc = subprocess.call(command, stdout=out, stderr=subprocess.OUTPUT)

Of course you can also temporarily wrap stdout to write to the logger and just pass the output through, e.g., using this confusingly identically-named class:

with StreamLogger('stdout'):
    proc = subprocess.call(command, stderr=subprocess.OUTPUT)