Cron job on a large database with PHP, suggestions needed

Question

I've heard about cron job and don't think the actual creation of it will be that hard to make but I've some concerns about how this will work with a large script.

Without going too much off-topic on my project i will stick with the basics about my situation. I need to make a script that every day performs a CURL fetch for data on a remote website and updates an database for each featured member on my website with it. In short, it's approximatively at this time 1000 times the script need to be executed but it will be a larger number as times goes by.

As you can guess, this will take a long time to preform so i'm worried about how the execution will work in a manner of not crashing in the middle of it.

My first thought was to perhaps split the users into groups and make the executions on a small amount of users each time but don't know how this is manageable ( will read on further about the topic when i got some form of confirmation on this).

So, to my question. Do you think there is any way for me to make this happen and do you perhaps have any suggestions on how to make this to work efficiently? All help i can get is appreciated. Thank you for your time.

Answer 1

bigger cron-jobs with php and mysql needs to be fragmented, since there is no way for you to 'nice' them, (reduce their os priority). Even if you nice the script, the mysql-requests will be executed without this concern.

From what you're describing there's two aspects to consider:

• Congestion of network bandwith
• Congestion of database throughput

I'd recommend a fragmented solution where you call your script from cron more often, and let the script execute only a small amount of the total job. The job should further be canceled (postponed to next run) if i/o-bandwith or cpu-usage is above any limit that may affect response-time to visitors.

regards, /t