Checking for duplicate cron jobs

Question

We are deploying code to our application server environment, and part of that process is creating a number of cron jobs on the server. When code gets pushed, our deployment script creates the required cron jobs without a problem using the following:

CRON_FILE=$SCRIPT_DIR/cron.txt 
if [[ ! -f "$CRON_FILE" ]]; then
        printf "Cron template file missing!\n\n"
        exit 1
fi
while read LINE || [[ -n "$LINE" ]]; do
        printf "\n> Adding cron job \"$LINE\"\n"
        crontab -l | { cat; echo "$LINE"; } | crontab -
done < $CRON_FILE

The issue is that after the initial deployment, additional deployments are creating duplicate cron jobs.

Any pointers on how to detect if a cron job already exists?

Answer 1

When you add your cron job, include a comment with a unique label. Later you can use that unique label to determine if the cron job exists or not, and also to "uninstall" the cron job.

I do this all the time. I have a reusable script for this:

#!/bin/sh
#
# Usage: 
# 1. Put this script somewhere in your project
# 2. Edit "$0".crontab file, it should look like this, 
#    but without the # in front of the lines
#0  *   *   *   *   stuff_you_want_to_do
#15 */5 *   *   *   stuff_you_want_to_do
#*  *   1,2 *   *   and_so_on
# 3. To install the crontab, simply run the script
# 4. To remove the crontab, run ./crontab.sh --remove
# 

cd $(dirname "$0")

test "$1" = --remove && mode=remove || mode=add

cron_unique_label="# $PWD"

crontab="$0".crontab
crontab_bak=$crontab.bak
test -f $crontab || cp $crontab.sample $crontab

crontab_exists() {
    crontab -l 2>/dev/null | grep -x "$cron_unique_label" >/dev/null 2>/dev/null
}

# if crontab is executable
if type crontab >/dev/null 2>/dev/null; then
    if test $mode = add; then
        if ! crontab_exists; then
            crontab -l > $crontab_bak
            echo 'Appending to crontab:'
            cat $crontab
            crontab -l 2>/dev/null | { cat; echo; echo $cron_unique_label; cat $crontab; echo; } | crontab -
        else
            echo 'Crontab entry already exists, skipping ...'
            echo
        fi
        echo "To remove previously added crontab entry, run: $0 --remove"
        echo
    elif test $mode = remove; then
        if crontab_exists; then
            echo Removing crontab entry ...
            crontab -l 2>/dev/null | sed -e "\?^$cron_unique_label\$?,/^\$/ d" | crontab -
        else
            echo Crontab entry does not exist, nothing to do.
        fi
    fi
fi

Save the script as crontab.sh in your project directory, and create a crontab.sh.crontab with your cron job definitions, for example:

0 0 * * * echo hello world
0 0 * * * date

• To install your cron jobs, simply run ./crontab.sh
• The script is safe to run multiple times: it will detect if the unique label already exists and skip adding your cron jobs again
• To uninstall the cron jobs, run ./crontab.sh --remove

I put this on GitHub too: https://github.com/janosgyerik/crontab-script

Explanation of sed -e "\?^$cron_unique_label\$?,/^\$/ d":

• In its simplest form the expression is basically: sed -e '/start/,/end/ d'
• It means: delete the content between the lines matching the start pattern and the end pattern, including the lines containing the patterns
• The script quotes the sed command with double-quotes instead of single quotes, because it needs to expand the value of the $cron_unique_label shell variable
• The start pattern \?^$cron_unique_label\$? uses a pair of ? instead of / to enclose the pattern, because $cron_unique_label contains /, which would cause problems
• The starting ? must be escaped with a backslash, but to be honest I don't know why.
• The ^ matches start of the line and $ end of the line, and the $ must be escaped, otherwise the shell would expand the value of the $? shell variable
• The end pattern /^\$/ is relatively simple, it matches a start of line followed by end of line, in other words an empty line, and again the $ must be escaped
• The d at the end is the sed command, to delete the matched lines, effectively removing it from the content of crontab -l, which we can pipe to crontab -

Answer 2

Weird, but very thorough answers. IMO overly complex.

Here's a decent one liner for anyone coming to this for a 2017 answer:

crontab -l | grep 'match-your-cronjob-search' || (crontab -l 2>/dev/null; echo "* * * * * /bin/cronjobCommandYouWant >> /dev/null 2>&1") | crontab -

Works great for us to not have dupe crons.

And here's the edited script from the original poster:

    CRON_FILE=$SCRIPT_DIR/cron.txt 
if [[ ! -f "$CRON_FILE" ]]; then
        printf "Cron template file missing!\n\n"
        exit 1
fi
while read LINE || [[ -n "$LINE" ]]; do
        printf "\n> Adding cron job \"$LINE\"\n"
        crontab -l | grep "$LINE" || (crontab -l 2>/dev/null; echo "$LINE") | crontab -
done < $CRON_FILE

Answer 3

Your script janos is awesome, works perfectly and was exactly what i was looking for with one little glitch. I couldnt manage multiple xxx.crontab templates. Your script worked fine and i added it to my bootstrapping routines with a little modification so i can pass a first parameter $1 of the xx.crontab filename and second parameter $2 can be the removal. I have parent shell scripts which then conditional decide, which, all or combination of crontab files i want to add/remove.

Here the script with my modifications included:

#!/bin/sh
#
# Usage:
# 1. Put this script somewhere in your project
# 2. Edit "$1".crontab file, it should look like this,
#    but without the # in front of the lines
#0  *   *   *   *   stuff_you_want_to_do
#15 */5 *   *   *   stuff_you_want_to_do
#*  *   1,2 *   *   and_so_on
# 3. To install the crontab, run ./crontab.sh <nameOfCronTabfile>
# 4. To remove the crontab, run ./crontab.sh <nameOfCronTabfile> --remove

cd $(dirname "$0")

test "$2" = --remove && mode=remove || mode=add

cron_unique_label="# cmID:$PWD|$1#"

crontab="$1".crontab
crontab_bak=$crontab.bak
test -f $crontab || cp $crontab.sample $crontab

crontab_exists() {
    crontab -l 2>/dev/null | grep -x "$cron_unique_label" >/dev/null 2>/dev/null
}

# if crontab is executable
if type crontab >/dev/null 2>/dev/null; then
    if test $mode = add; then
        if ! crontab_exists; then
            crontab -l > $crontab_bak
            echo 'Appending to crontab:'
            echo '-----------------------------------------------'
            cat $crontab
            crontab -l 2>/dev/null | { cat; echo; echo $cron_unique_label; cat $crontab; echo "# cm #"; } | crontab -
        else
            echo 'Crontab entry already exists, skipping ...'
            echo
        fi
        echo '-----------------------------------------------'
        echo "To remove previously added crontab entry, run: $0 $1 --remove"
        echo
    elif test $mode = remove; then
        if crontab_exists; then
            echo 'Removing crontab entry ...'
            crontab -l 2>/dev/null | sed -e "\?^$cron_unique_label\$?,/^# cm #\$/ d" | crontab -
        else
            echo 'Crontab entry does not exist, nothing to do.'
        fi
    fi
fi

UPDATE: Sry, didnt noticed the weak empty line pattern for removing a crontab. Simply would delete everything after the found crontab id, including manually added crontabs. Changed the empty line end pattern to a little end tag. So it will add crontabs with:

crontab -l 2>/dev/null | { cat; echo; echo $cron_unique_label; cat $crontab; echo "# cm #"; } | crontab -

.. and removing exactly only this cron with:

crontab -l 2>/dev/null | sed -e "\?^$cron_unique_label\$?,/^# cm #\$/ d" | crontab -