Is there any way to have output piped line-by-line from a currently executing python program?

Question

When piping printed output from a python script to a command like grep, the output from the script seems to only be piped to the follow-up command after completion of the entire script.

For example, in a script test_grep.py like the following:

#!/usr/bin/env python
from time import sleep

print "message1"
sleep(5)
print "message2"
sleep(5)
print "message3"

when called with ./test_grep.py | grep message, nothing will appear for 10 seconds, at which time all three lines will appear.

Compare this to a script test_grep.sh:

#!/usr/bin/env bash
echo "message1"
sleep 5 
echo "message2"
sleep 5
echo "message3"

./test_grep.sh | grep message will immediately output message1, followed at 5 second intervals by message2 and message3.

I expect this is because only once the python interpreter finishes executing is the output available for the next command. Is there any way to alter this behavior?

Answer 1

You can do it:

• By flushing every print in python
• By setting stdout to be unbuffered
• By setting stdout to be line-buffered

You can even call python -u to disable buffering.

---

I would go for the line-buffering option as it seems most natural.

open(file, mode='r', buffering=-1 ....)

buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size of a fixed-size chunk buffer.

When you don't specify buffering (the typical "open") it will use line-buffering if it detects the output is going directly do a TTY, i.e. to your screen console. If you pipe output or redirect it to a file it will switch back to a large (4K / 8K) buffer.

---

How do you "set stdout to be line-buffered"?

You can reopen stdout via sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1).