Streaming shell output from Flask view works, but never ends

Question

I want to stream the output of several python3 scripts to my browser. I followed some suggestions from different SO answers and nearly solved it. But my for loop reading from stdout runs into an infinite loop after executing the script. The output is correct and everything is fine, but the endless loop is a problem. How can I end the stream after the output is finished?

@app.route('/stream/<script>')
def execute(script):
    def inner():
        assert re.match(r'^[a-zA-Z._-]+$', script)
        exec_path = "scripts/" + script + ".py"
        cmd = ["python3", "-u", exec_path]  # -u: don't buffer output

        proc = subprocess.Popen(
            cmd,
            stdout=subprocess.PIPE,
        )

        for line in iter(proc.stdout.readline, ''):
            yield highlight(line, BashLexer(), HtmlFormatter())
            # If process is done, break loop
   #         if proc.poll() == 0:
   #             break

    env = Environment(loader=FileSystemLoader('app/templates'))
    tmpl = env.get_template('stream.html')
    return Response(tmpl.generate(result=inner()))

When I poll the subprocess to see if it has finished, the output of the script is broken, because it does not print the entire stdout stream, when the prints come too fast. If I add a sleep(1) between each print, the problem does not occur.

The code works, but runs into an infinite loop. If I uncomment these lines, the output is somehow buffered and parts of it get lost. I also tried to poll the subprocess in a separate thread and just have to toggle a flag like if not running: break, but this leads to the same behaviour like the uncommented code above.

Answer 1

iter(proc.stdout.readline, '') calls readline until it encounters something that equals ''. But proc.stdout.readline returns bytes objects, and '' is a str object, so the two will never equal!

Instead, either write

for line in iter(proc.stdout.readline, b''):

or, even better:

for line in proc.stdout: