Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Risks of running a cronjob every minute

Tags:

php

mysql

xml

cron

I have a php script designed to check a certain folder for xml files and then import the information from each file to a MySQL database.

I would like to set up a cronjob to run every minute so that anytime new files are added they will almost instantly be imported without me having to manually ssh in and run the script.

I have an if statement which checks if files exist and only runs the code if they do, otherwise 'No files' is echoed.

I would like to know if there are any risks to having this run constantly, will excessive resources be taken up? etc

like image 999
Edward144 Avatar asked Dec 14 '17 14:12

Edward144


People also ask

How often can you run a cron job?

Unfortunately cronjobs can run only at a maximum of once per minute. So in the worst-case a user has to wait one minute until his Email is really going to be sent. My current idea for a workaround is calling the script with an addtional sleep parameter and duplicating the cronjobs.

How long can a cron job run?

Cron is a daemon, a long-running process that only needs to be started once, and will run constantly in the background. Cron wakes up every minute, examines its list of things to do to see if any scheduled tasks need to be executed, and if so it executes them. If not, it goes back to sleep for another 59 seconds.

Which is the correct cron job that runs every 5 minutes?

The utility used for scheduling these jobs is called crontab. A common cron job that Linux admins use on their systems is to execute a command or script every 5 minutes.


1 Answers

There's really nothing wrong with running a process every minute ... except the usual pitfalls [which I include with ways to mitigate]. I do want to say that a minute is a really really long time for a modern computer. If you are short of cycles, a few extra system calls per minute is the wrong place to look.

  • pitfall #1 is that something goes "wrong" with the script and for some reason it doesn't exit. Symptom: box crashes as it can't create anymore processes and/or open a file descriptor, etc.

How to solve: make the script grab an exclusive lock to a file. You could write your pid to a file, but that's hacky. If you can't grab the exclusive lock, a previous version is running, so you should just exit.

Here's the PHP interface to flock(): PHP flock()

  • pitfall #2: it really should be a daemon.

If something needs to be "done all the time", maybe it should really be "done all the time". You can use the file locking recipe to make sure your script stays up, or you can use something like monit to start it. But you can also just insure it stays up by using cron, and file locking.

  • Pitfall #3: you convert to a daemon, but there's a memory leak and the thing just keeps expanding up like the girl in Willie Wonka on too many blueberries. Symptom: OOM error, swapping, etc. This is PHP after all.

Solution: exit after 1000 [or some #] iterations, and then use cron and the file locking model to start a new version [or monit or equivalent].

like image 195
Bret Weinraub Avatar answered Sep 20 '22 20:09

Bret Weinraub