PHP memory usage in For loop keeps growing

Question

I have a script that I run which does a lot of tasks and goes through about 21k times. The problem is for each index I am doing several different things, each index is a product in our database, I am updating the price by grabbing data from a API and saving the product and so on. I have several areas where I have put calls to memory_get_usage() before and after almost every method call, and everyone that I do have seems to increase the memory. There isn't one that is doing more than the others or its not that noticable.

I have tried to unset all my variables at the bottom of the loop as well as trying to just set them to null, but no matter what the memory limit just keeps raising through each iteration.

Is there anything I can do to clear this memory up, I would think unsetting the variables were suppose to free up the memory but it doesn't seem to do so?

EDIT: I forgot to mention the reason why I started to investigate this is that I am getting memory limit errors on the server. It doesn't always happen at the same point or even happen everytime it is ran. That is why I tried to investigate it.

The script takes about an hour to run, I run it in the morning when nothing else is going on and right now it is only on a staging server anyway so there really isn't anyone hitting the server.

I can post the code but its pretty big

<?php


if( !function_exists('memory_get_usage') ){
    include('function.php');
}
echo "At the start we're using (in bytes): ",
     memory_get_usage() , "\n\n";

$path = realpath(dirname(__FILE__) . '/../../../../Mage.php');
require_once($path);
Mage::app();
require_once '/lib/ProductUpdate.php';
echo "Starting product update process \n\n";
$productUpdate = new ProductUpdate();
$dealerStoreId = 3;
$volumeDiscountGroupId = 4;
$retailGroupId = Mage_Customer_Model_Group::CUST_GROUP_ALL;
$wholesaleGroupId = 2;


echo "Grabbing all products \n\n";
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

// get the products from the InOrder stored procedure qty since datetime and don't pass a date to get all products, also pass the id of the cron job
$ioProducts = $productUpdate->getProductUpdateProducts('WEB2');
echo "---------------------------\n\n";
echo "Begin Updating Products \n\n";
echo "---------------------------\n\n";
$productCount = 0;
$productUpdate->saveScriptStarted(2);
echo "Before we go into the initial loop we are using (in bytes): ",
     memory_get_usage() , "\n\n";
foreach ($ioProducts as $ioProduct) {
    $updateProduct = false;
    $updateTierPrice = false;
    $sku = trim($ioProduct['inp_short_item_number']) . trim($ioProduct['isc_SIZE']) . trim($ioProduct['isc_COLOR']);
    echo "Checking item number " . $sku . " \n\n";
    echo "Before Loading Product " . $sku .  " we are using (in bytes): ",
     memory_get_usage() , "\n\n";
    $product = $productUpdate->getProduct();
    $productId = $product->getIdBySku($sku);
    echo "After Getting Id from sku " . $sku .  " we are using (in bytes): ",
     memory_get_usage() , "\n\n";
    if ($productId) {
        //$product = $productUpdate->getProduct()->load($productId);
        echo "After Loading Product " . $sku .  " we are using (in bytes): ",
            memory_get_usage() , "\n\n";
        echo "WE HAVE A PRODUCT!: " . $product->getName() . "\n\n";

        try {
            echo "Before Getting Additional Info from InOrder for Product " . $sku .  " we are using (in bytes): ",
            memory_get_usage() , "\n\n";

            // Since the product is same for parent products as it is for children you should just be able to get the price  of the parent and use that.
            $additionalInfo = $productUpdate->getItemDetails($ioProduct['inp_short_item_number'], 'WEB2');

            echo "After Getting Additional Info from InOrder for Product " . $sku .  " we are using (in bytes): ",
            memory_get_usage() , "\n\n";

            echo "Before Getting Extra Charges from InOrder for Product " . $sku .  " we are using (in bytes): ",
            memory_get_usage() , "\n\n";

            $oversizeCharge = $productUpdate->getExtraCharges($ioProduct['inp_short_item_number']);

            echo "After Getting Extra Charges from InOrder for Product " . $sku .  " we are using (in bytes): ",
            memory_get_usage() , "\n\n";
        } catch (Exception $e) {
            echo $e->getMessage() . "\n\n";
            continue;
        }

        if (is_array($additionalInfo) && count($additionalInfo) > 0) {
            if (isset($oversizeCharge[0]['Shipping Unit Charge']) && $product->getOversizeCharge() != $oversizeCharge[0]['Shipping Unit Charge']) {
                $product->setOversizeCharge($oversizeCharge[0]['Shipping Unit Charge']);
                $updateProduct = true;
                unset($oversizeCharge);
            }
            if ($product->getPrice() != $additionalInfo[0]['pri_current_price']) {
                $product->setPrice($additionalInfo[0]['pri_current_price']);
                $updateProduct = true;
                unset($additionalInfo);
            }
            echo "Before Setting Stock Status for Product " . $sku .  " we are using (in bytes): ",
            memory_get_usage() , "\n\n";

            $product = $productUpdate->setStockStatus($product, $ioProduct);

            echo "After Setting Stock Status for Product " . $sku .  " we are using (in bytes): ",
            memory_get_usage() , "\n\n";

            if ($product->getNeedsUpdate()) {
                $updateProduct = true;
            }

            if ($updateProduct) {
                try{
                    echo "Before Saving Product " . $sku .  " we are using (in bytes): ",
                    memory_get_usage() , "\n\n";

                   $productUpdate->saveProduct($product, $ioProduct);

                    echo "After Saving Product " . $sku .  " we are using (in bytes): ",
                    memory_get_usage() , "\n\n";
                }catch (Exception $e){
                    echo $e->getMessage() . "\n\n";
                    continue;
                }
            }


            // Go through  and do the same thing for the other 2 web classes to set pricing for the Dealer and Volume wholesale customers
            $updateProduct = false;
            try {
                echo "Before getting Tier Price info for " . $sku .  " we are using (in bytes): ",
                memory_get_usage() , "\n\n";

                $product = $productUpdate->getProduct()->setStoreId($dealerStoreId)->load($productId);
                $additionalInfo = $productUpdate->getItemDetails($ioProduct['inp_short_item_number'], 'WEB3');
                //$additionalTierInfo = $productUpdate->getItemDetails($ioProduct['inp_short_item_number'], 'WEB4');

                // Get Real Tier Prices based on Customer Type
                $retailPriceBreaks = $productUpdate->getItemPriceBreaks($ioProduct['inp_short_item_number'], Mage::getStoreConfig(Lancaster_InOrder_Helper_Data::XML_PATH_RETAIL_PRICE_LIST));
                $wholesalePriceBreaks = $productUpdate->getItemPriceBreaks($ioProduct['inp_short_item_number'], Mage::getStoreConfig(Lancaster_InOrder_Helper_Data::XML_PATH_WHOLESALE_PRICE_LIST));
                $volumeWholesalePriceBreaks = $productUpdate->getItemPriceBreaks($ioProduct['inp_short_item_number'], Mage::getStoreConfig(Lancaster_InOrder_Helper_Data::XML_PATH_VOLUME_WHOLESALE_PRICE_LIST));

                echo "After getting Tier Price infor for " . $sku .  " we are using (in bytes): ",
                memory_get_usage() , "\n\n";
            } catch (Exception $e) {
                echo $e->getMessage() . "\n\n";
                continue;
            }


            if ($product->getPrice() != $additionalInfo[0]['pri_current_price']) {
                $product->setPrice($additionalInfo[0]['pri_current_price']);
                $updateProduct = true;
            }

            //The only way to setup multiple price for one website is to set a tier price so we set it to a specific group and the dealer site then go through and set all the other real tier prices
            $tierPriceInfo = $product->getData('tier_price');
            if (!empty($tierPriceInfo)) {
                echo "Before looping through Tier Price infor for " . $sku .  " we are using (in bytes): ",
                memory_get_usage() , "\n\n";
                foreach ($tierPriceInfo as $tierPrice) {
                    if ($tierPrice["website_id"] == $dealerStoreId &&
                        $tierPrice["cust_group"] == $volumeDiscountGroupId &&
                        $tierPrice["price_qty"] == '1' &&
                        $tierPrice["price"] != $additionalTierInfo[0]['pri_current_price']) {
                        $updateTierPrice = true;
                    }
                    //todo need to do some refinement to the following, was rushed to put out the logic need to fix so it doesn't update everytime
                    // need to find if any of the tier prices do not match price as well if there is a price break in InOrder but not in Magento

                    if (!$updateTierPrice ) {

                        $updateRetail = isUpdateTierPrices($retailPriceBreaks, $tierPrice, $retailGroupId);
                        $updateWholesale = isUpdateTierPrices($wholesalePriceBreaks, $tierPrice, $wholesaleGroupId);
                        $updateVolWholesale = isUpdateTierPrices($volumeWholesalePriceBreaks, $tierPrice, $volumeDiscountGroupId);
                        if (
                            (count($retailPriceBreaks) > 0 && !$updateRetail['priceTierExists']) &&
                            (count($wholesalePriceBreaks) > 0 && !$updateWholesale['priceTierExists']) &&
                            (count($volumeWholesalePriceBreaks) > 0 && !$updateVolWholesale['priceTierExists'])) {
                             $updateTierPrice = true;
                        }

                        if(($updateRetail['updateTierPrice'] || $updateWholesale['updateTierPrice'] || $updateVolWholesale['updateTierPrice'])){
                            $updateTierPrice = true;
                        }
                    }
                }
                unset($tierPriceInfo);
                echo "After looping through Tier Price infor for " . $sku .  " we are using (in bytes): ",
                memory_get_usage() , "\n\n";
            }
            else {
                $updateTierPrice = true;
            }
            if ($updateTierPrice) {
                echo "Before setting whether we update Tier Price for " . $sku .  " we are using (in bytes): ",
                memory_get_usage() , "\n\n";
                //construct the tier price
                $website_id = Mage::getModel('core/store')->load($dealerStoreId)->getWebsiteId();
                $tierPrices = array(array(
                                        'website_id' => $website_id,
                                        'cust_group' => $volumeDiscountGroupId,
                                        'price_qty' => '1',
                                        'price' => $additionalTierInfo[0]['pri_current_price']
                                    ));

                updateTierPrices($retailPriceBreaks, $retailGroupId, $tierPrices);
                updateTierPrices($wholesalePriceBreaks, $wholesaleGroupId, $tierPrices);
                updateTierPrices($volumeWholesalePriceBreaks, $volumeDiscountGroupId, $tierPrices);

                $product->setData('tier_price', $tierPrices);
                $updateProduct = true;
                unset($website_id);
                echo "After setting whether we update Tier Price for " . $sku .  " we are using (in bytes): ",
                memory_get_usage() , "\n\n";
            }

            if ($updateProduct) {
                try{
                     echo "Before saving product for Tiered Pricing for " . $sku .  " we are using (in bytes): ",
                    memory_get_usage() , "\n\n";
                  // $productUpdate->saveProduct($product, $ioProduct);
                    echo "After saving product for Tiered Pricing for " . $sku .  " we are using (in bytes): ",
                    memory_get_usage() , "\n\n";
                }catch (Exception $e){
                    echo $e->getMessage() . "\n\n";
                    continue;
                }

            }
        }
    }
    $retailPriceBreaks = null;
    $wholesalePriceBreaks = null;
    $volumeWholesalePriceBreaks = null;
    $oversizeCharge = null;
    $additionalTierInfo = null;
    $additionalInfo = null;
    $product = null;
    $productCount++;
    echo $productCount . " Products have been proceessed \n\n";
}
echo "After running through all products we are using (in bytes): ",
                    memory_get_usage() , "\n\n";
echo "Peak memory usage for product update scrip (in bytes): ",
                    memory_get_peak_usage() , "\n\n";

Answer 1

Increasing memory usage in PHP is normal. unsetting a variable doesn't immediately free up the memory it was taking, it simply marks it as available to be re-used. At some point, PHP'll decide the garbage collector should be run, and that's when the memory is truly freed.

Unless you're actually running into "out of memory" fatal errors, this is nothing to worry about. PHP does its best to prevent OOM from happening, but it's not going to do very expensive garbage collection runs every time you unset a variable. Performance would literally grind to a halt if that was happening.