Read console output of another program in Python

Question

There are two programs - parent.py and myscript.py, shown below. parent.py keeps printing messages in the console. And myscript.py needs access to what parent.py prints.

parent.py:

import time

while 1:
     time.sleep(1)
     print "test"

myscript.py:

import subprocess
p = None

p = subprocess.Popen(['python', 'parent.py'],
                 stdout=subprocess.PIPE,
                 stderr=subprocess.STDOUT,
                 )
for line in p.stdout:
       print line

I want myscript.py to catch the output of parent.py dynamically. As it can be seen that parent.py has an infinite loop - the script is never gonna stop, but whenever it outputs the word 'test', myscript.py should also output the same. But myscript.py doesn't give out any output.

I tried replacing the code in parent.py with just

print "Hello"

Since the program finished executing, myscript.py also printed "Hello". So I guess unless the execution of parent.py is complete, myscript.py doesn't read it. Is that whats happening?

I also tried replacing the loop in myscript.py with this snippet:

for line in iter(p.stdout.readline, ''):
    print line

Still, no output.

My question is: How do I get myscript.py dynamically read all that is being printed by parent.py even when parent.py is never going to complete its execution.

(I found similar questions, but they either didn't work or there were no answers at all in some of them.)

Answer 1

It is a block buffering issue: when stdout is redirected to a pipe parent.py accumulates its output in an internal buffer (~4K-8K) i.e., you won't see anything for around an hour until the buffer overflows.

To disable buffering, pass -u command-line option:

#!/usr/bin/env python
import os
import sys
from subprocess import Popen, PIPE, STDOUT

script_path = os.path.join(get_script_dir(), 'parent.py')
p = Popen([sys.executable, '-u', script_path],
          stdout=PIPE, stderr=STDOUT, bufsize=1)
with p.stdout:
    for line in iter(p.stdout.readline, b''):
        print line,
p.wait()

See:

• Python: read streaming input from subprocess.communicate()
• Call python script with input with in a python script using subprocess

Note: if you don't need to do anything with the output then just drop stdout=PIPE, to see the output in the console:

subprocess.check_call([sys.executable, script_path])