How to prevent BrokenPipeError when doing a flush in Python?

Question

Question: Is there a way to use flush=True for the print() function without getting the BrokenPipeError?

I have a script pipe.py:

for i in range(4000):
    print(i)

I call it like this from a Unix command line:

python3 pipe.py | head -n3000

And it returns:

0
1
2

So does this script:

import sys
for i in range(4000):
    print(i)
    sys.stdout.flush()

However, when I run this script and pipe it to head -n3000:

for i in range(4000):
    print(i, flush=True)

Then I get this error:

    print(i, flush=True)
BrokenPipeError: [Errno 32] Broken pipe
Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored

I have also tried the solution below, but I still get the BrokenPipeError:

import sys
for i in range(4000):
    try:
        print(i, flush=True)
    except BrokenPipeError:
        sys.exit()

Answer 1

The BrokenPipeError is normal as said phantom because the reading process (head) terminates and closes its end of the pipe while the writing process (python) still tries to write.

Is is an abnormal condition, and the python scripts receives a BrokenPipeError - more exactly, the Python interpreter receives a system SIGPIPE signal that it catches and raises the BrokenPipeError to allow the script to process the error.

And you effectively can process the error, because in your last example, you only see a message saying that the exception was ignored - ok it is not true, but seems related to this open issue in Python : Python developpers think important to warn user of the abnormal condition.

What really happens is that AFAIK the python interpreter always signals this on stderr, even if you catch the exception. But you just have to close stderr before exiting to get rid of the message.

I slightly changed your script to :

• catch the error as you did in your last example
• catch either IOError (that I get in Python34 on Windows64) or BrokenPipeError (in Python 33 on FreeBSD 9.0) - and display a message for that
• display a custom Done message on stderr (stdout is closed due to the broken pipe)
• close stderr before exiting to get rid of the message

Here is the script I used :

import sys

try:
    for i in range(4000):
            print(i, flush=True)
except (BrokenPipeError, IOError):
    print ('BrokenPipeError caught', file = sys.stderr)

print ('Done', file=sys.stderr)
sys.stderr.close()

and here the result of python3.3 pipe.py | head -10 :

0
1
2
3
4
5
6
7
8
9
BrokenPipeError caught
Done

If you do not want the extraneous messages just use :

import sys

try:
    for i in range(4000):
            print(i, flush=True)
except (BrokenPipeError, IOError):
    pass

sys.stderr.close()

Answer 2

A note on SIGPIPE was added in Python 3.7 documentation, and it recommends to catch BrokenPipeError this way:

import os
import sys

def main():
    try:
        # simulate large output (your code replaces this loop)
        for x in range(10000):
            print("y")
        # flush output here to force SIGPIPE to be triggered
        # while inside this try block.
        sys.stdout.flush()
    except BrokenPipeError:
        # Python flushes standard streams on exit; redirect remaining output
        # to devnull to avoid another BrokenPipeError at shutdown
        devnull = os.open(os.devnull, os.O_WRONLY)
        os.dup2(devnull, sys.stdout.fileno())
        sys.exit(1)  # Python exits with error code 1 on EPIPE

if __name__ == '__main__':
    main()

Importantly, it says:

Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.

Answer 3

According to the Python documentation, this is thrown when:

trying to write on a pipe while the other end has been closed

This is due to the fact that the head utility reads from stdout, then promptly closes it.

As you can see, it can be worked around by merely adding a sys.stdout.flush() after every print(). Note that this sometimes does not work in Python 3.

You can alternatively pipe it to awk like this to get the same result as head -3:

python3 0to3.py | awk 'NR >= 4 {exit} 1'

Hope this helped, good luck!