Python: subprocess32 process.stdout.readline() waiting time

Question

If I run the following function "run" with for example "ls -Rlah /" I get output immediately via the print statement as expected

import subprocess32 as subprocess
def run(command):
    process = subprocess.Popen(command,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.STDOUT)
    try:
        while process.poll() == None:
            print process.stdout.readline()
    finally:
        # Handle the scenario if the parent
        # process has terminated before this subprocess
        if process.poll():
            process.kill()

However if I use the python example program below it seems to be stuck on either process.poll() or process.stdout.readline() until the program has finished. I think it is stdout.readline() since if I increase the number of strings to output from 10 to 10000 (in the example program) or add in a sys.stdout.flush() just after every print, the print in the run function does get executed.

How can I make the output from a subprocess more real-timeish?

Note: I have just discovered that the python example program does not perform a sys.stdout.flush() when it outputs, is there a way for the caller of subprocess to enforce this somehow?

Example program which outputs 10 strings every 5 seconds.

#!/bin/env python
import time

if __name__ == "__main__":

    i = 0
    start = time.time()
    while True:
        if time.time() - start >= 5:
            for _ in range(10):
                print "hello world" + str(i)
            start = time.time()
            i += 1
        if i >= 3:
            break

Answer 1

On most systems, command line programs line buffer or block buffer depending on whether stdout is a terminal or a pipe. On unixy systems, the parent process can create a pseudo-terminal to get terminal-like behavior even though the child isn't really run from a terminal. You can use the pty module to create a pseudo-terminal or use the pexpect module which eases access to interactive programs.

As mentioned in comments, using poll to read lines can result in lost data. One example is data left in the stdout pipe when the process terminates. Reading pty is a bit different than pipes and you'll find you need to catch an IOError when the child closes to get it all to work properly as in the example below.

try:
    import subprocess32 as subprocess
except ImportError:
    import subprocess
import pty
import sys
import os
import time
import errno

print("running %s" % sys.argv[1])

m,s = (os.fdopen(pipe) for pipe in pty.openpty())
process = subprocess.Popen([sys.argv[1]],
                           stdin=s,
                           stdout=s,
                           stderr=subprocess.STDOUT)
s.close()

try:
    graceful = False
    while True:
        line = m.readline()
        print line.rstrip()
except IOError, e:
    if e.errno != errno.EIO:
        raise
    graceful = True
finally:
    # Handle the scenario if the parent
    # process has terminated before this subprocess
    m.close()
    if not graceful:
        process.kill()
    process.wait()

Answer 2

You should flush standard output in your script:

print "hello world" + str(i)
sys.stdout.flush()

When standard output is a terminal, stdout is line-buffered. But when it is not, stdout is block buffered and you need to flush it explicitly.

If you can't change the source of your script, you can use the -u option of Python (in the subprocess):

-u     Force stdin, stdout and stderr to be totally unbuffered. 

Your command should be: ['python', '-u', 'script.py']

In general, this kind of buffering happens in userspace. There are no generic ways to force an application to flush its buffers: some applications support command line options (like Python), others support signals, others do not support anything.

One solution might be to emulate a pseudo terminal, giving "hints" to the programs that they should operate in line-buffered mode. Still, this is not a solution that works in every case.