Getting output of a process at runtime

Question

I am using a python script to run a process using subprocess.Popen and simultaneously store the output in a text file as well as print it on the console. This is my code:

result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in result.stdout.readlines(): #read and store result in log file
    openfile.write("%s\n" %line)
    print("%s" %line)

Above code works fine, but what it does is it first completes the process and stores the output in result variable. After that for loop stores the output as well as print it.

But i want the output at runtime (as my process can take hours to complete, i don't get any output for all these hours).

So is there any other function that gives me the output dynamically (at runtime), means as soon as the process gives first line, it should get printed.

Answer 1

.readlines() returns a list of all the lines the process will return while open, i.e., it doesn't return anything until all output from the subprocess is received. To read line by line in "real time":

import sys
from subprocess import Popen, PIPE

proc = Popen(cmd, shell=True, bufsize=1, stdout=PIPE)
for line in proc.stdout:
    openfile.write(line)
    sys.stdout.buffer.write(line)
    sys.stdout.buffer.flush()
proc.stdout.close()
proc.wait()

Note: if the subprocess uses block-buffering when it is run in non-interactive mode; you might need pexpect, pty modules or stdbuf, unbuffer, script commands.

Note: on Python 2, you might also need to use iter(), to get "real time" output:

for line in iter(proc.stdout.readline, ""):
    openfile.write(line)
    print line,

Answer 2

The problem here is that .readlines() gets the entire output before returning, as it constructs a full list. Just iterate directly:

for line in result.stdout:
    print line

Answer 3

You can iterate over the lines one by one by using readline on the pipe:

while True: 
    line = result.stdout.readline()
    print line.strip()
    if not line:
        break

The lines contain a trailing \n which I stripped for printing. When the process terminates, readline returns an empty string, so you know when to stop.