Reliably kill sleep process after USR1 signal

Question

I am writing a shell script which performs a task periodically and on receiving a USR1 signal from another process.

The structure of the script is similar to this answer:

#!/bin/bash

trap 'echo "doing some work"' SIGUSR1

while :
do
    sleep 10 && echo "doing some work" &
    wait $!
done

However, this script has the problem that the sleep process continues in the background and only dies on its timeout. (note that when USR1 is received during wait $!, the sleep process lingers for its regular timeout, but the periodic echo indeed gets cancelled.) You can for example see the number of sleep processes on your machine using pkill -0 -c sleep.

I read this page, which suggests killing the lingering sleep in the trap action, e.g.

#!/bin/bash

pid=
trap '[[ $pid ]] && kill $pid; echo "doing some work"' SIGUSR1

while :
do
    sleep 10 && echo "doing some work" &
    pid=$!
    wait $pid
    pid=
done

However this script has a race condition if we spam our USR1 signal fast e.g. with:

pkill -USR1 trap-test.sh; pkill -USR1 trap-test.sh

then it will try to kill a PID which was already killed and print an error. Not to mention, I do not like this code.

Is there a better way to reliably kill the forked process when interrupted? Or an alternative structure to achieve the same functionality?

Answer 1

As the background job is a fork of the foreground one, they share the same name (trap-test.sh); so pkill matches and signals both. This, in an uncertain order, kills the background process (leaving sleep alive, explained below) and triggers the trap in the foreground one, hence the race condition.

Besides, in the examples you linked, the background job is always a mere sleep x, but in your script it is sleep 10 && echo 'doing some work'; which requires the forked subshell to wait sleep to terminate and conditionally execute echo. Compare these two:

$ sleep 10 &
[1] 9401
$ pstree 9401
sleep
$
$ sleep 10 && echo foo &
[2] 9410
$ pstree 9410
bash───sleep

So let's start from scratch and reproduce the principal issue in a terminal.

$ set +m
$ sleep 100 && echo 'doing some work' &
[1] 9923
$ pstree -pg $$
bash(9871,9871)─┬─bash(9923,9871)───sleep(9924,9871)
                └─pstree(9927,9871)
$ kill $!
$ pgrep sleep
9924
$ pkill -e sleep
sleep killed (pid 9924)

I disabled job control to partly emulate a non-interactive shell's behavior.

Killing the background job didn't kill sleep, I needed to terminate it manually. This happened because a signal sent to a process is not automatically broadcast to its target's children; i.e. sleep didn't receive the TERM signal at all.

To kill sleep as well as the subshell, I need to put the background job into a separate process group —which requires job control to be enabled, otherwise all jobs are put into the main shell's process group as seen in pstree's output above—, and send the TERM signal to it, as shown below.

$ set -m
$ sleep 100 && echo 'doing some work' &
[1] 10058
$ pstree -pg $$
bash(9871,9871)─┬─bash(10058,10058)───sleep(10059,10058)
                └─pstree(10067,10067)
$ kill -- -$!
$
[1]+  Terminated              sleep 100 && echo 'doing some work'
$ pgrep sleep
$

With some refinement and adaptation of this concept, your script looks like:

#!/bin/bash -
set -m

usr1_handler() {
  kill -- -$!
  echo 'doing some work'
}

do_something() {
  trap '' USR1
  sleep 10 && echo 'doing some work'
}

trap usr1_handler USR1 EXIT

echo "my PID is $$"

while true; do
  do_something &
  wait
done

This will print my PID is xxx (where xxx is the PID of foreground process) and start looping. Sending a USR1 signal to xxx (i.e kill -USR1 xxx) will trigger the trap and cause the background process and its children to terminate. Thus wait will return and the loop will continue.

If you use pkill instead it'll work anyway, as the background process ignores USR1.

For further information, see:

• Bash Reference Manual § Special Parameters ($$ and $!),
• POSIX kill specification (-$! usage),
• POSIX Definitions § Job Control (how job control is implemented in POSIX shells),
• Bash Reference Manual § Job Control Basics (how job control works in bash),
• POSIX Shell Command Language § Signals And Error Handling,
• POSIX wait specification.