python stream subprocess stdout and stderr zip doesnt work

Question

There are already a bunch of answers that solve how to do this general thing, but my main question is Why doesnt this approach work?

I am trying to "live stream" the stdout and stderr from a subprocess. I can do this by doing:

import sys
import subprocess

def run_command(cmd):
    process = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)

    for out in iter(process.stdout.readline, b''):
        print(out)

    for err in iter(process.stderr.readline, b''):
        print(err)

run_command(['echo', 'hello', 'world']) # should print hello world
run_command(['rm', 'blarg223'])  # Should print to stderr (file doesnt exist)

and this works giving me the result:

b'hello world\n'
b'rm: cannot remove \xe2\x80\x98blarg223\xe2\x80\x99: No such file or directory\n'

However this causes a problem as it really only live streams the stdout, then prints all the errors as the end. So I thought I could fix this using:

def run_command(cmd):
    process = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)

    for out, err in zip(iter(process.stdout.readline, b''), iter(process.stderr.readline, b'')):
        print(out)
        print(b'Error: ' + err)

But, this doesnt produce any output. Why does using zip not work?

Answer 1

zip stops when one of the iterators is finished.

In each of the examples you gave, one stream(stdout/stderr) is empty. So zip will produce nothing.

To fix this you should use itertools.zip_longest