Why does my Python3 script balk at piping its output to head or tail (sys module)?

Question

I have a Python3 script that writes its output to stdout, but it complains when I pipe that output into head or tail. Note in the sample output below that it sort of works, in that head is returning the first two lines of output as requested.

> ./script.py '../Testdata/*indels.ss' -m 5 | head -2                                                                                              ~/Databases/Avian_genomes/Sandbox/combined
xread
2999 50
Traceback (most recent call last):
  File "./new.py", line 194, in <module>
    sys.stdout.write(lineout)
IOError: [Errno 32] Broken pipe
Exception IOError: IOError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored

In contrast, the script has no problem with its output being piped to awk as below.

> ./script.py '../Testdata/*indels.ss' -m 5 | awk 'NR < 3 {print $0}'                                                                              ~/Databases/Avian_genomes/Sandbox/combined
xread
2999 50

Let me know if you need some code from the script beyond what's contained in the error message. I'm not sure what would be relevant.

Answer 1

./script.py '../Testdata/*indels.ss' -m 5 | awk 'NR >= 3 {exit} 1'

would show the same behavior as head -2.

You can turn set the SIGPIPE handler to one which quietly kills your program instead:

import signal
signal.signal(signal.SIGPIPE, signal.SIG_DFL)

Answer 2

I'll cite from here:

If a sequence of commands appears in a pipeline, and one of the

reading commands finishes before the writer has finished, the

writer receives a SIGPIPE signal.

That's what head does. Your script hasn't finished writing, but head is already done, so stdout is closed, hence the exception.