stdin read python from piped looping process

Question

How can I spawn or Popen a subprocess in python and process its output in realtime? The subprocess prints output randomly depending on other system events.

This "example" hangs:

$./print.sh | ./echo.py hangs.

print.sh

#!/bin/bash

while [ 1 ]; do
echo 'A'
sleep 1
done

echo.py

#!/usr/bin/python
import sys

for line in sys.stdin:
  print line

Answer 1

It doesn't hang. echo/the shell decides that, because it's writing to a pipe, it will perform I/O in a block-buffered rather than line-buffered mode. If you wait long enough, or remove the sleep 1 from the shell script, you'll see that the output from A does come through.

There are two possible solutions:

1. Modify the subprocess's program so that it flushes its buffers when it's written enough output for the Python program to process.
2. Use pseudo-terminals (PTYs) instead of pipes. pexpect does that, hiding most of the complexity from you. It's not a drop-in replacement for subprocess, though.