Reading stdout process in real time

Question

Given this code snippet:

from subprocess import Popen, PIPE, CalledProcessError


def execute(cmd):
    with Popen(cmd, shell=True, stdout=PIPE, bufsize=1, universal_newlines=True) as p:
        for line in p.stdout:
            print(line, end='')

    if p.returncode != 0:
        raise CalledProcessError(p.returncode, p.args)

base_cmd = [
    "cmd", "/c", "d:\\virtual_envs\\py362_32\\Scripts\\activate",
    "&&"
]
cmd1 = " ".join(base_cmd + ['python -c "import sys; print(sys.version)"'])
cmd2 = " ".join(base_cmd + ["python -m http.server"])

If I run execute(cmd1) the output will be printed without any problems.

However, If I run execute(cmd2) instead nothing will be printed, why is that and how can I fix it so I could see the http.server's output in real time.

Also, how for line in p.stdout is been evaluated internally? is it some sort of endless loop till reaches stdout eof or something?

This topic has already been addressed few times here in SO but I haven't found a windows solution. The above snippet is code from this answer and I'm running http.server from a virtualenv (python3.6.2-32bits on win7)

Answer 1

If you want to read continuously from a running subprocess, you have to make that process' output unbuffered. Your subprocess being a Python program, this can be done by passing -u to the interpreter:

python -u -m http.server

This is how it looks on a Windows box.

Answer 2

With this code, you can`t see the real-time output because of buffering:

for line in p.stdout:
    print(line, end='')

But if you use p.stdout.readline() it should work:

while True:
  line = p.stdout.readline()
  if not line: break
  print(line, end='')

See corresponding python bug discussion for details

UPD: here you can find almost the same problem with various solutions on stackoverflow.