How to retry a command in Bash?

Question

I have a command that should take less than 1 minute to execute, but for some reason has an extremely long built-in timeout mechanism. I want some bash that does the following:

success = False

try(my_command)

while(!(success))
wait 1 min
if my command not finished
     retry(my_command)
else
     success = True   
end while

How can I do this in Bash?

Answer 1

I found a script from: http://fahdshariff.blogspot.com/2014/02/retrying-commands-in-shell-scripts.html

#!/bin/bash
 
# Retries a command on failure.
# $1 - the max number of attempts
# $2... - the command to run

retry() {
    local -r -i max_attempts="$1"; shift
    local -i attempt_num=1
    until "$@"
    do
        if ((attempt_num==max_attempts))
        then
            echo "Attempt $attempt_num failed and there are no more attempts left!"
            return 1
        else
            echo "Attempt $attempt_num failed! Trying again in $attempt_num seconds..."
            sleep $((attempt_num++))
        fi
    done
}
 
# example usage:
retry 5 ls -ltr foo

Answer 2

Look at the GNU timeout command. This kills the process if it has not completed in a given time; you'd simply wrap a loop around this to wait for the timeout to complete successfully, with delays between retries as appropriate, etc.

while timeout -k 70 60 -- my_command; [ $? = 124 ]
do sleep 2  # Pause before retry
done

If you must do it in pure bash (which is not really feasible - bash uses lots of other commands), then you are in for a world of pain and frustration with signal handlers and all sorts of issues.

---

Please expand on your answer a little. -k 70 is --kill-after= 70 seconds, 124 exit on timeout; what is the 60?

The linked documentation does explain the command; I don't really plan to repeat it all here. The synopsis is timeout [options] duration command [arg]...; one of the options is -k duration. The -k duration says "if the command does not die after the SIGTERM signal is sent at 60 seconds, send a SIGKILL signal at 70 seconds" (and the command should die then). There are a number of documented exit statuses; 124 indicates that the command timed out; 137 that it died after being sent the SIGKILL signal, and so on. You can't tell if the command itself exits with one of the documented statuses.

Answer 3

I liked @Jonathan's answer, but tried to make it more straight forward for future use:

until timeout 1 sleep 2
do
    echo "Happening after 1s of sleep"
done

Answer 4

Adapting @Shin's answer to use kill -0 rather than jobs so that this should work even with classic Bourne shell, and allow for other background jobs. You may have to experiment with kill and wait depending on how my_command responds to those.

while true ; do
    my_command &
    sleep 60
    if kill -0 $! 2>/dev/null; then
        # Job took too long
        kill $!
    else
        echo "Job is done"
        # Reap exit status
        wait $!
        break
    fi
done