immediate piping with python

Question

I have two files first of which is fizz

#!/usr/bin/python                                                         
import time

print 'started'
time.sleep(3)
print 'ended'

next of which is bar

#!/usr/bin/python                                                         
import sys

for line in sys.stdin:
    print line

When I run the command ./fizz | ./bar I expect it to print started then wait 3 seconds and print ended, but what really happends is that it prints started and ended at the same time after 3 seconds. Is there a way to get my desired behavior? Thanks

Answer 1

Now that it is clear that the problem is on the receiving side, I present an alternative which I like to use:

#!/usr/bin/python                                                   
import sys 
import os

for line in iter(sys.stdin.readline, ''):
    sys.stdout.write(line) # \n included in line

iter(func, sentinel) calls func() for each iteration and ends if the function result == sentinel.

Answer 2

Good question. It is a little harder than it should.

The issue is indeed in bar, specifically sys.stdin being buffered. I tried opening sys.stdin with a smaller buffer size and using python -u but that didn't work. The manpage has this to say:

   -u     Force  stdin,  stdout  and  stderr to be totally unbuffered.  On
          systems where it matters, also put stdin, stdout and  stderr  in
          binary  mode.   Note  that there is internal buffering in xread‐
          lines(), readlines() and file-object  iterators  ("for  line  in
          sys.stdin")  which  is  not  influenced by this option.  To work
          around this, you will want to use "sys.stdin.readline()"  inside
          a "while 1:" loop.

In the end this is what worked for me:

#!/usr/bin/python                                                   
import sys 
import os

while True:
    line = sys.stdin.readline()
    if not line:
        break
    sys.stdout.write(line) # or print, doesn't matter.

Answer 3

There are two problems:

1. print "something" in ./foo doesn't flush its stdout buffer if it is redirected (to the pipe in this case) i.e., when stdout is not connected a tty-like device e.g., to an interactive console
2. for line in sys.stdin: might try to read several lines at a time

You can fix it as follows:

$ PYTHONUNBUFFERED=1 ./foo | ./bar

Where ./bar:

#!/usr/bin/python
import sys

for line in iter(sys.stdin.readline, ''):
    print line,

i.e., make ./foo's stdout unbuffered (-u option) and read input in ./bar line by line as suggested in @Eduardo Ivanec's answer.

As alternative you could call sys.stdout.flush() in ./foo instead of making its stdout unbuffered as suggested in @kev's answer.