Broken-pipe Error Python subprocess [duplicate]

Question

I'm trying to launch several bash routines from a GUI based software. The problem I'm facing is a piping issue. Here the test bash-script (bashScriptTest.sh):

#!/bin/bash
#---------- Working
ls | sort | grep d > testFile.txt
cat testFile.txt
#---------- NOT working
echo $RANDOM > testFile2.txt
for i in `seq 1 15000`; do
    echo $RANDOM >> testFile2.txt
done
awk '{print $1}' testFile2.txt | sort -g | head -1

And here the python script that creates the error:

import subprocess
#
with open('log.txt','w') as outfile:
    CLEAN=subprocess.Popen("./bashScriptTest.sh", stdout=outfile, stderr=outfile)
    print CLEAN.pid
    OUTSEE=subprocess.Popen(['x-terminal-emulator', '-e','tail -f '+outfile.name])

As you can see from running the python script, the Broken-pipe error is encountered not in the first three pipes (first line) but instead after the huge work done by awk. I need to manage an huge quantities of routine and subroutines in bash and also using the shell==True flag doesn't change a thing. I tried to write everything in the most pythonic way but unfortunately there is no chance I can rewrite all the piping step inside python. Another thing to mention is that if you test the bash script inside a terminal everything works fine. Any help would be really appreciated. Thanks in advance!

EDIT 1:

The log file containing the error says:

bashScriptTest.sh
log.txt
stack.txt
testFile2.txt
test.py
3
sort: write failed: standard output: Broken pipe
sort: write error

Answer 1

Okay so this is a little bit obscure, but it just so happens that I ran across a similar issue while researching a question on the python-tutor mailing list some time ago.

The reason you're seeing different behavior when running your script via the subprocess module (in python) vs. bash directly, is that python overrides the disposition of SIGPIPEs to SIG_IGN (ignore) for all child processes (globally).

When the following pipeline is executed ...

awk '{print $1}' testFile2.txt | sort -g | head -1

... head will exit after it prints the first line of stdout from the sort command, due to the -1 flag. When the sort command attempts to write more lines to its stdout, a SIGPIPE is raised.

The default action of a SIGPIPE; when the pipeline is executed in a shell like bash, for example; is to terminate the sort command.

As stated earlier, python overrides the default action with SIG_IGN (ignore), so we end up with this bizarre, and somewhat inexplicable, behavior.

---

That's all well and good, but you might be wondering what to do now? It's dependant on the version of python you're using ...

For Python 3.2 and greater, you're already set. subprocess.Popen in 3.2 added the restore_signals parameter, which defaults to True, and effectively solves the issue without further action.

For previous versions, you can supply a callable to the preexec_fn argument of subprocess.Popen, as in ...

import signal
def default_sigpipe():
    signal.signal(signal.SIGPIPE, signal.SIG_DFL)

# ...

with open('log.txt','w') as outfile:
    CLEAN=subprocess.Popen("./bashScriptTest.sh", 
                           stdout=outfile, stderr=outfile
                           preexec_fn=default_sigpipe)

I hope that helps!

EDIT: It should probably be noted that your program is actually functioning properly, AFAICT, as is. You're just seeing additional error messages that you wouldn't normally see when executing the script in a shell directly (for the reasons stated above).

See Also:

• https://mail.python.org/pipermail/python-dev/2007-July/073831.html
• https://bugs.python.org/issue1652