wget not executed through cronjob but directly in console

Question

I have an TP Link Archer C7 with 15.05 Chaos Calmer at home.

I am trying to get the cronjob to work. It looks like this:

* * * * * /usr/bin/wget --spider \"https://subdomain.domain.net/update.php\" >/dev/null 2>&1

Unfortunately this does not get executed. I also tried to do* * * * wget -O - https://subdomain.domain.net/update.php same result. BOTH versions are executed correctly if entered through the console directly.

Last line of logread

Tue Sep 20 11:55:00 2016 cron.info crond[10317]: USER xxx pid 16697 cmd /usr/bin/wget --spider \"https://subdomain.domain.net/update.php\" >/dev/null 2>&1

Looks correct? Any idea why it is not executed through the cronjob?

Answer 1

Your cron job does run, but wget runs into an error.

Also to get a log, i.s.o. redirecting to /dev/null, you cannot use /out.txt, as cron will have no write access to /, /tmp/wget.txt will do fine.

Same goes for the wget command, where does the output get stored? you should use use -p <directory prefix> option.

Secondly you need to handle if data is already in that directory, which could be to use -N and only download newer files and override the old ones.

Thirdly use the -q option when all is working to quiet down wget output.

Fourthly you cannot run the script every minute * * * * *, as previous run might not have finished yet, see format:

 # ┌───────────── min (0 - 59)
 # │ ┌────────────── hour (0 - 23)
 # │ │ ┌─────────────── day of month (1 - 31)
 # │ │ │ ┌──────────────── month (1 - 12)
 # │ │ │ │ ┌───────────────── day of week (0 - 6) (0 to 6 are Sunday to
 # │ │ │ │ │                  Saturday, or use names; 7 is also Sunday)
 # │ │ │ │ │
 # │ │ │ │ │
 # * * * * *  command to execute

For the sake of test let's run it every night at 1 past midnight.

Remark: To really make it correct, you should use a script file to execute and put wget command in there, where as the script file could make a lock in case you want to execute this frequently, see here for advice as this is beyond the scope of this question/answer.

And Number 5: don't need to escape the " with \", even so the " is not needed, so remove it.

In conclusion, make first a directory where data can be stored e.g. /wget_data

mkdir /wget_data
chmod 777 /wget_data

then adjust your cron line, add the -p <directory prefix> and -N and adjust the execute format:

1 0 * * *  /usr/bin/wget -N -p /wget_data --spider https://subdomain.domain.net/update.php >/tmp/wget.txt 2>&1

and restart your cron, can't advice on that, don't known what is used yet.

also wise would be to to test your command from the command line first with:

/usr/bin/wget -N -p /wget_data --spider https://subdomain.domain.net/update.php >/tmp/wget.txt 2>&1

and then cat /tmp/wget.txt to see if the command worked, and ls /wget_data to see if wget did collect data.