Kill next command in pipeline on failure

Question

I have a streaming backup script which I'm running as follows:

./backup_script.sh | aws s3 cp - s3://bucket/path/to/backup

The aws command streams stdin to cloud storage in an atomic way. If the process is interrupted without an EOF, the upload is aborted.

I want the aws process to be killed if ./backup_script.sh exits with a non-zero exit code.

Any bash trick for doing this?

EDIT: You can test your solution with this script:

#!/usr/bin/env python
import signal
import sys
import functools

def signal_handler(signame, signum, frame):
    print "Got {}".format(signame)
    sys.exit(0)

signal.signal(signal.SIGTERM, functools.partial(signal_handler, 'TERM'))
signal.signal(signal.SIGINT, functools.partial(signal_handler, 'INT'))

for i in sys.stdin:
    pass

print "Got EOF"

Example:

$ grep --bla | ./sigoreof.py
grep: unrecognized option `--bla'
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
    [-e pattern] [-f file] [--binary-files=value] [--color=when]
    [--context[=num]] [--directories=action] [--label] [--line-buffered]
    [--null] [pattern] [file ...]
Got EOF

I want ./sigoreof.py to be terminated with a signal.

Answer 1

backup_script.sh should have a non-zero exit status if there is an error, so you script should look something like:

if ./backup_script.sh > output.txt; then
    aws s3 cp output.txt s3://bucket/path/to/backup
fi
rm -f output.txt

A pipe isn't really appropriate here.

---

If you really need to conserve disk space locally, you'll have to "reverse" the upload; either remove the uploaded file in the event of an error in backup_script.sh, or upload to a temporary location, then move that to the final path once you've determined that the backup has succeeded.

(For simplicity, I'm ignoring the fact that by letting aws exit on its own in the event of an error, you may be uploading more of the partial backup than you need to. See Charles Duffy's answer for a more bandwidth-efficient approach.)

After starting the backup process with

mkfifo data
./backup_script.sh > data & writer_pid=$!

use one of the following to upload the data.

# Upload and remove if there was an error
aws s3 cp - s3://bucket/path/to/backup < data &

if ! wait $writer_pid; then
    aws s3 rm s3://bucket/path/to/backup
fi

or

# Upload to a temporary file and move it into place
# once you know the backup succeeded.
aws s3 cp - s3://bucket/path/to/backup.tmp < data &

if wait $writer_pid; then
    aws s3 mv s3://bucket/path/to/backup.tmp s3://bucket/path/to/backup
else
    aws s3 rm s3://bucket/path/to/backup
fi

Answer 2

Adopting/correcting a solution originally given by @Dummy00001:

mkfifo aws.fifo
exec 3<>aws.fifo # open the FIFO read/write *in the shell itself*
aws s3 cp - s3://bucket/path/to/backup <aws.fifo 3>&- & aws_pid=$!
rm aws.fifo # everyone who needs a handle already has one; can remove the directory entry

if ./backup_script.sh >&3 3>&-; then
    exec 3>&-       # success: close the FIFO and let AWS exit successfully
    wait "$aws_pid"
else
    kill "$aws_pid" # send a SIGTERM...
    wait "$aws_pid" # wait for the process to die...
    exec 3>&-       # only close the write end *after* the process is dead
fi

---

Important points:

• The shell opens the FIFO r/w to avoid blocking (opening for write only would block for a reader; this could also be avoided by invoking the reader [that is, the s3 command] in the background prior to the exec opening the write side).
• The write end of the FIFO is held by the script itself, so the read end never hits end-of-file until after the script intentionally closes it.
• The aws command's handle on the write end of the FIFO is explicitly closed (3<&-), so it doesn't hold itself open (in which case the exec 3>&- done in the parent would not successfully allow it to finish reading and exit).

Answer 3

A short script which uses process substitution instead of named pipes would be:

#!/bin/bash

exec 4> >( ./second-process.sh )
./first-process.sh >&4  &
if ! wait $! ; then echo "error in first process" >&2; kill 0; wait; fi

It works much like with a fifo, basically using the fd as the information carrier for the IPC instead of a file name.

Two remarks: I wasn't sure whether it's necessary to close fd 4 ; I would assume that upon script exit the shell closes all open files.

And I couldn't figure out how to obtain the PID of the process in the process substitution (anybody? at least on my cygwin the usual $! didn't work.) Therefore I resorted to killing all processes in the group, which may not be desirable (but I'm not entirely sure about the semantics).