Calling python script with subprocess.Popen and flushing the data

Question

Ok so i've seen dozen of threads like that , but none of them gives a complete answer and everything i tried so far foes not work for me.

1) Script that constantly outputs some data and flusheshs it:

import time
import sys

if __name__ == '__main__':
    for i in range(5):
        print i,
        sys.stdout.flush()
        time.sleep(1)

2) Script that calls first script with Popen and should be printing numbers one by one but for some reason does not, and prints them alltogether at once :

import sys
import subprocess

if __name__ == '__main__':
    process = subprocess.Popen(['python', 'flush.py'], stdout = subprocess.PIPE )
    for line in iter(process.stdout.readline, ''):
        print line,
        sys.stdout.flush()

First thing i am a little bit confused is in the first script is that if you remove the flush it returns output in one line alltogether O_O... I am pretty sure it is because of time.sleep but still kind of expected it return like a standart output constantly returning values 0,1,2,3,4 but not all together, ofcourse flush resolves it , but just strange, at least for me ...

The main problem: Is that second script does not return number one by one , but returns all in one output at once..... What i need is to see numbers popping one by one...

I read somewhere that it does not return EOF which Popen waits to close the pipe , thats why it runs like to the end .....

So what do i do or try next ? Thanks in advance.

Answer 1

As @Warren Weckesser's comment says, your problem is unrelated to buffering issues.

.readline() in the parent process won't return until it reads a newline or reaches EOF. Your child process doesn't print any newlines at all so your parent process doesn't print anything until the child process ends.

The minimal fix is just to remove comma at the end of print i, in the child script.

This also works:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

p = Popen([sys.executable or 'python',
           '-u', # unbuffer stdout (or make it line-buffered on Python 3)
           '-c',
           """
import time

for i in range(5):
    print(i) # <-- no comma i.e., each number is on its own line
    time.sleep(1)
"""], stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
    print(int(line)**2)

Example:

 $ python parent.py
 0
 1
 4
 9
 16

The numbers are printed every seconds without waiting for the child process to end.

If you don't want to change the child script then you should use readline() that stops at whitespace instead of a newline character e.g.:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

p = Popen(['python2', 'child.py'], stdout=PIPE, bufsize=0)
for token in generate_tokens(p.stdout):
    print(int(token))

where generate_tokens() yields whitespace-separated tokens:

def generate_tokens(pipe):
    buf = []
    while True:
        b = pipe.read(1) # read one byte
        if not b: # EOF
            pipe.close()
            if buf:
                yield b''.join(buf)
            return
        elif not b.isspace(): # grow token
            buf.append(b)
        elif buf: # full token read
            yield b''.join(buf)
            buf = []

It also prints integers as soon as they are printed by the child.