Possible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script

Question

UPDATE: While not actually solving the original problem presented with regards to my piping endeavours, I've solved my problem by simplifying it greatly, and just ditching the pipes altogether. Here's a proof-of-concept script that generates, in parallel while reading just once from disk, CRC32, MD5, SHA1, SHA224, SHA256, SHA384, and SHA512 checksums, and returning them as a JSON object (gonna use the output in PHP and Ruby). It's crude without error checking, but it works:

#!/bin/bash

checksums="`tee <"$1" \
        >( cfv -C -q -t sfv -f - - | tail -n 1 | sed -e 's/^.* \([a-fA-F0-9]\{8\}\)$/"crc32":"\1"/' ) \
        >( md5sum - | sed -e 's/^\([a-fA-F0-9]\{32\}\) .*$/"md5":"\1"/' ) \
        >( sha1sum - | sed -e 's/^\([a-fA-F0-9]\{40\}\) .*$/"sha1":"\1"/' ) \
        >( sha224sum - | sed -e 's/^\([a-fA-F0-9]\{56\}\) .*$/"sha224":"\1"/' ) \
        >( sha256sum - | sed -e 's/^\([a-fA-F0-9]\{64\}\) .*$/"sha256":"\1"/' ) \
        >( sha384sum - | sed -e 's/^\([a-fA-F0-9]\{96\}\) .*$/"sha384":"\1"/' ) \
        >( sha512sum - | sed -e 's/^\([a-fA-F0-9]\{128\}\) .*$/"sha512":"\1"/') \
        >/dev/null`\ 
"

json="{"

for checksum in $checksums; do json="$json$checksum,"; done

echo "${json:0: -1}}"

THE ORIGINAL QUESTION:

I'm a bit afraid to ask this question, as I got so many hits on my search phrase that after applying the knowledge harvested from Using named pipes with bash - Problem with data loss, and reading through another 20 pages, I still am at a bit of a stand-still with this.

So, to continue nevertheless, I'm doing a simple script to enable me to concurrently create CRC32, MD5, and SHA1 checksums on a file while only reading it from disk once. I'm using cfv for that purpose.

Originally, I just hacked together a simply script that wrote cat'ted the file to tee with three cfv commands writing to three separate files under /tmp/, and then attempted to cat them out to stdout afterwards, but ended up with empty output unless I made my script sleep for a second before trying to read the files. Thinking that was weird, I assumed I was a moron in my scripting, so I tried to do a different approach by having the cfv workers output to a named pipe instead. So far, this is my script, after having applied techniques from forementioned link:

!/bin/bash

# Bail out if argument isn't a file:
[ ! -f "$1" ] && echo "'$1' is not a file!" && exit 1

# Choose a name for a pipe to stuff with CFV output:
pipe="/tmp/pipe.chksms"

# Don't leave an orphaned pipe on exiting or being terminated:
trap "rm -f $pipe; exit" EXIT TERM

# Create the pipe (except if it already exists (e.g. SIGKILL'ed b4)):
[ -p "$pipe" ] || mkfifo $pipe

# Start a background process that reads from the pipe and echoes what it
# receives to stdout (notice the pipe is attached last, at done):
while true; do
        while read line; do
                [ "$line" = "EOP" ] && echo "quitting now" && exit 0
                echo "$line"
        done
done <$pipe 3>$pipe & # This 3> business is to make sure there's always
                      # at least one producer attached to the pipe (the
                      # consumer loop itself) until we're done.

# This sort of works without "hacks", but tail errors out when the pipe is
# killed, naturally, and script seems to "hang" until I press enter after,
# which I believe is actually EOF to tail, so it's no solution anyway:
#tail -f $pipe &

tee <"$1" >( cfv -C -t sfv -f - - >$pipe ) >( cfv -C -t sha1 -f - - >$pipe ) >( cfv -C -t md5 -f - - >$pipe ) >/dev/null

#sleep 1s
echo "EOP" >$pipe
exit

So, executed as it stands, I get this output:

daniel@lnxsrv:~/tisso$ ./multisfv file
 :  :  : quitting now
- : Broken pipe (CF)
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
- : Broken pipe (CF)
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
- : Broken pipe (CF)
daniel@lnxsrv:~/tisso$ close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

But, with the sleep 1s commented out, I the get expected output,

daniel@lnxsrv:~/tisso$ ./multisfv file
3bc1b5ff125e03fb35491e7d67014a3e *
-: 1 files, 1 OK.  0.013 seconds, 79311.7K/s
5e3bb0e3ec410a8d8e14fef1a6daababfc48c7ce *
-: 1 files, 1 OK.  0.016 seconds, 62455.0K/s
; Generated by cfv v1.18.3 on 2012-03-09 at 23:45.23
;
2a0feb38
-: 1 files, 1 OK.  0.051 seconds, 20012.9K/s
quitting now

This puzzles me, as I'd assume that tee wouldn't exit until after each cfv recipient it forks data to has exited, and thus the echo "EOP" statement would execute until all cfv substreams have finished, which would mean they'd have written their output to my named pipe... And then the echo statement would execute.

As the behavior is the same without pipes, just using output temporary files, I'm thinking this must be some race condition having to do with the way tee pushes data onto its recipients? I tried a simple "wait" command, but it'll of course wait for my bash child process - the while loop - to finish, so I just get a hanging process.

Any ideas?

TIA, Daniel :)

Answer 1

tee will exit once it writes the last bit of input to the last output pipe and closes it (that is, the unnamed pipes created by bash, not your fifo, aka "named pipe"). It has no need to wait for the processes reading the pipes to finish; indeed, it has no idea that it is even writing to pipes. Since pipes have buffers, it's quite likely that tee finishes writing before the processes at the other end have finished reading. So the script will write 'EOP' into the fifo, causing the read loop to terminate. That will close the fifo's only reader, and all the cfv processes will get SIGPIPE when they next try to write to stdout.

The obvious question to ask here is why you don't just run three (or N) independent processes reading the file and computing different summaries. If "the file" were actually being generated on the fly or downloaded from some remote site, or some other slow process, it might make sense to do things the way you're trying to do them, but if the file is present on local disk, it's pretty likely that only one disk access will actually happen; the lagging summarizers will read the file from the buffer cache. If that's all you need, GNU parallel should work fine, or you can just start up the processes in bash (with &) and then wait for them. YMMV but I would think that either of these solutions will be less resource-intensive than setting up all those pipes and simulating the buffer cache in userland with tee.

By the way, if you want to serialize output from multiple processes, you can use the flock utility. Just using a fifo isn't enough; there is no guarantee that the processes writing to the fifo will write entire lines atomically and if you knew they did that, you wouldn't need the fifo.