Long PHP script runs multiple times

Question

I have a products database that synchronizes with product data ever morning.

The process is very clear:

• Get all products from database by query
• Loop through all products, and get and xml from the other server by product_id
• Update data from xml
• Log the changes to file.

If I query a low amount of items, but limiting it to 500 random products for example, everything goes fine. But when I query all products, my script SOMETIMES goes on the fritz and starts looping multiple times. Hours later I still see my log file growing and products being added.

I checked everything I could think of, for example:

• Are variables not used twice without overwriting each other
• Does the function call itself
• Does it happen with a low amount of products too: no.
• The script is called using a cronjob, are the settings ok. (Yes)

The reason that makes it especially weird is that it sometimes goes right, and sometimes it doesnt. Could this be some memory problem?

EDIT wget -q -O /dev/null http://example.eu/xxxxx/cron.php?operation=sync its in webmin called on a specific hour and minute

Code is hundreds of lines long...

Thanks

Answer 1

You have:

• max_execution_time disabled. Your script won't end until the process is complete for as long as it needed.
• memory_limit disabled. There is no limit to how much data stored in memory.

500 records were completed without issues. This indicates that the scripts completes its process before the next cronjob iteration. For example, if your cron runs every hour, then the 500 records are processed in less than an hour.

If you have a cronjob that is going to process large amount of records, then consider adding lock mechanism to the process. Only allow the script to run once, and start again when the previous process is complete.

You can create script lock as part of a shell script before executing your php script. Or, if you don't have an access to your server you can use database lock within the php script, something like this.

class ProductCronJob
{
    protected $lockValue;

    public function run()
    {
        // Obtain a lock
        if ($this->obtainLock()) {
            // Run your script if you have valid lock
            $this->syncProducts();

            // Release the lock on complete
            $this->releaseLock();
        }
    }

    protected function syncProducts()
    {
        // your long running script
    }

    protected function obtainLock()
    {
        $time = new \DateTime;
        $timestamp = $time->getTimestamp();
        $this->lockValue = $timestamp . '_syncProducts';

        $db = JFactory::getDbo();

        $lock = [
            'lock'         => $this->lockValue,
            'timemodified' => $timestamp
        ];
        // lock = '0' indicate that the cronjob is not active.
        // Update #__cronlock set lock = '', timemodified = '' where name = 'syncProducts' and lock = '0'
//        $result = $db->updateObject('#__cronlock', $lock, 'id');

//        $lock = SELECT * FROM #__cronlock where name = 'syncProducts';

        if ($lock !== false && (string)$lock !== (string)$this->lockValue) {
            // Currently there is an active process - can't start a new one

            return false;

            // You can return false as above or add extra logic as below

            // Check the current lock age - how long its been running for
//            $diff = $timestamp - $lock['timemodified'];
//            if ($diff >= 25200) {
//                // The current script is active for 7 hours.
//                // You can change 25200 to any number of seconds you want.
//                // Here you can send notification email to site administrator.
//                // ...
//            }
        }

        return true;
    }

    protected function releaseLock()
    {
        // Update #__cronlock set lock = '0' where name = 'syncProducts'
    }
}