real time subprocess.Popen via stdout and PIPE

Question

I am trying to grab stdout from a subprocess.Popen call and although I am achieving this easily by doing:

cmd = subprocess.Popen('ls -l', shell=True, stdout=PIPE)
for line in cmd.stdout.readlines():
    print line

I would like to grab stdout in "real time". With the above method, PIPE is waiting to grab all the stdout and then it returns.

So for logging purposes, this doesn't meet my requirements (e.g. "see" what is going on while it happens).

Is there a way to get line by line, stdout while is running? Or is this a limitation of subprocess(having to wait until the PIPE closes).

EDIT If I switch readlines() for readline() I only get the last line of the stdout (not ideal):

In [75]: cmd = Popen('ls -l', shell=True, stdout=PIPE)
In [76]: for i in cmd.stdout.readline(): print i
....: 
t
o
t
a
l

1
0
4

Answer 1

Your interpreter is buffering. Add a call to sys.stdout.flush() after your print statement.

Answer 2

Actually, the real solution is to directly redirect the stdout of the subprocess to the stdout of your process.

Indeed, with your solution, you can only print stdout, and not stderr, for instance, at the same time.

import sys
from subprocess import Popen
Popen("./slow_cmd_output.sh", stdout=sys.stdout, stderr=sys.stderr).communicate()

The communicate() is so to make the call blocking until the end of the subprocess, else it would directly go to the next line and your program might terminate before the subprocess (although the redirection to your stdout will still work, even after your python script has closed, I tested it).

That way, for instance, you are redirecting both stdout and stderr, and in absolute real time.

For instance, in my case I tested with this script slow_cmd_output.sh:

#!/bin/bash

for i in 1 2 3 4 5 6; do sleep 5 && echo "${i}th output" && echo "err output num ${i}" >&2; done