Python: Capture stdout from subprocess.call

Question

I'm trying to do two things when executing a shell cmd with Python:

• Capture stdout and print it as it happens
• Capture stdout as a whole and process it when the cmd is complete

I looked at subprocess.check_output, but it does not have an stdout param that would allow me to print the output as it happens.

So after reading this question, I realized I may need to try a different approach.

from subprocess import Popen, PIPE

process = Popen(task_cmd, stdout = PIPE)
stdout, stderr = process.communicate()

print(stdout, stderr)

The problem with this approach is that according to the docs, Popen.communicate():

Reads data from stdout and stderr, until end-of-file is reached. Wait for process to terminate

I still cannot seem to redirect output both to stdout AND to some sort of buffer that can be parsed when the command is complete.

Ideally, I'd like something like:

# captures the process output and dumps it to stdout in realtime
stdout_capture = Something(prints_to_stdout = True)
process = Popen(task_cmd, stdout = stdout_capture)

# prints the entire output of the executed process
print(stdout_capture.complete_capture)

Is there a recommended way to accomplish this?

Answer 1

You were on the right track with using giving Popen stdout=PIPE, but you can't use .communicate() because it returns the values after execution. Instead, I suggest you read from .stdout.

The only guaranteed way to get the output the moment it's generated is to read from the pipe one character at a time. Here is my approach:

def passthrough_and_capture_output(args):
    import sys
    import subprocess

    process = subprocess.Popen(args, stdout=subprocess.PIPE, universal_newlines=True)
    # universal_newlines means that the output of the process will be interpreted as text
    capture = ""

    s = process.stdout.read(1)
    while len(s) > 0:
        sys.stdout.write(s)
        sys.stdout.flush()
        capture += s
        s = process.stdout.read(1)

    return capture

Note that reading one character at a time can incur significant overhead, so if you are alright with lagging behind a bit, I suggest that you replace the 1 in read(1) with a different number of characters to output in batches.