Script works differently when ran from the terminal and ran from Python

Question

I have a short bash script foo.sh

#!/bin/bash

cat /dev/urandom | tr -dc 'a-z1-9' | fold -w 4 | head -n 1

When I run it directly from the shell, it runs fine, exiting when it is done

$ ./foo.sh 
m1un
$

but when I run it from Python

$ python -c "import subprocess; subprocess.call(['./foo.sh'])"
ygs9

it outputs the line but then just hangs forever. What is causing this discrepancy?

Answer 1

Adding the trap -p command to the bash script, stopping the hung python process and running ps shows what's going on:

$ cat foo.sh
#!/bin/bash

trap -p
cat /dev/urandom | tr -dc 'a-z1-9' | fold -w 4 | head -n 1

$ python -c "import subprocess; subprocess.call(['./foo.sh'])"
trap -- '' SIGPIPE
trap -- '' SIGXFSZ
ko5o

^Z
[1]+  Stopped     python -c "import subprocess; subprocess.call(['./foo.sh'])"
$ ps -H -o comm
COMMAND
bash
  python
    foo.sh
      cat
      tr
      fold
  ps

Thus, subprocess.call() executes the command with the SIGPIPE signal masked. When head does its job and exits, the remaining processes do not receive the broken pipe signal and do not terminate.

Having the explanation of the problem at hand, it was easy to find the bug in the python bugtracker, which turned out to be issue#1652.

Answer 2

The problem with Python 2 handling SIGPIPE in a non-standard way (i.e., being ignored) is already coined in Leon's answer, and the fix is given in the link: set SIGPIPE to default (SIG_DFL) with, e.g.,

import signal
signal.signal(signal.SIGPIPE,signal.SIG_DFL)

---

You can try to unset SIGPIPE from within your script with, e.g.,

#!/bin/bash

trap SIGPIPE # reset SIGPIPE

cat /dev/urandom | tr -dc 'a-z1-9' | fold -w 4 | head -n 1

but, unfortunately, it doesn't work, as per the Bash reference manual

Signals ignored upon entry to the shell cannot be trapped or reset.

---

A final comment: you have a useless use of cat here; it's better to write your script as:

#!/bin/bash

tr -dc 'a-z1-9' < /dev/urandom | fold -w 4 | head -n 1

Yet, since you're using Bash, you might as well use the read builtin as follows (this will advantageously replace fold and head):

#!/bin/bash

read -n4 a < <(tr -dc 'a-z1-9' < /dev/urandom)
printf '%s\n' "$a"

It turns out that with this version, you'll have a clear idea of what's going on (and the script will not hang):

$ python -c "import subprocess; subprocess.call(['./foo'])"
hcwh
tr: write error: Broken pipe
tr: write error
$
$ # script didn't hang

(Of course, it works well with no errors with Python3). And telling Python to use the default signal for SIGPIPE works well too:

$ python -c "import signal; import subprocess; signal.signal(signal.SIGPIPE,signal.SIG_DFL); subprocess.call(['./foo'])"
jc1p
$

(and also works with Python3).