run PHP script from cli : php.ini memory_size not taken into account

Question

I am trying to run a php script (magento reindexer script) from the command line. The script consumes a lot of memory so I am getting the following error: PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /home/karanta/www/karanta.fr/lib/Zend/Db/Adapter/Abstract.php on line 691

To fix this issue, I edited the /etc/php5/cli/php.ini file and set memory_limit = 2048M.

To check the configuration, I run a script containing phpinfo(); from the cli and I see: memory_limit => 2048M => 2048M, so it seems the configuration is taken into account properly. ini_get('memory_limit); also returns 2048M.

However when I rerun the reindex script, I still get PHP Fatal error: Allowed memory size of 536870912 bytes exhausted as if the memory_limit was still 512M. The script is unable to complete, I am out of ideas how to augment the memory_limit to allow the script to complete.

Edit: I also tried adding the instruction ini_set("memory_limit", -1); directly into the PHP script and it still hangs with the same PHP Fatal error: Allowed memory size of 536870912 bytes exhausted.

Extra information:

Server is a dedicated machine at ovh running Debian GNU/Linux 7.5 with 64GB RAM!

php -v returns:

PHP 5.6.12-1 (cli) Copyright (c) 1997-2015 The PHP Group Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies

free -k -h during script execution returns:

               total        used        free      shared  buff/cache     available
Mem:            62G        1,8G         48G         84M          12G         60G
Swap:          1,0G          0B        1,0G

ps -aux returns:

karanta  25568  5.6  0.0 229484 41824 pts/1    S    07:54   0:04 php shell/indexer.php --reindex catalog_url

---

Following @IgorGreg's recommandation, I tried setting the memory limit using Zend_Memory. I wrote this php script that I run from the cli in replacement of the shell/indexer.php script.

<?php
require_once 'app/Mage.php';
$app = Mage::app('admin');
umask(0);
$memoryManager = Zend_Memory::factory('none');
$memoryManager->setMemoryLimit(-1);
$process = Mage::getModel('index/process')->load(3);
$process->reindexAll();
?>

Still getting the same error.

Answer 1

Magento in CLI mode parses the .htaccess file and if it finds a memory limit, or any other PHP settings there, applies them (yes, seriously....)

The responsible code is in shell/abstract.php:

/**
 * Parse .htaccess file and apply php settings to shell script
 *
 */
protected function _applyPhpVariables()
{
    $htaccess = $this->_getRootPath() . '.htaccess';
    if (file_exists($htaccess)) {
        // parse htaccess file
        $data = file_get_contents($htaccess);
        $matches = array();
        preg_match_all('#^\s+?php_value\s+([a-z_]+)\s+(.+)$#siUm', $data, $matches, PREG_SET_ORDER);
        if ($matches) {
            foreach ($matches as $match) {
                @ini_set($match[1], str_replace("\r", '', $match[2]));
            }
        }
        preg_match_all('#^\s+?php_flag\s+([a-z_]+)\s+(.+)$#siUm', $data, $matches, PREG_SET_ORDER);
        if ($matches) {
            foreach ($matches as $match) {
                @ini_set($match[1], str_replace("\r", '', $match[2]));
            }
        }
    }
}

Solution

Do not set memory limit and execution time via .htaccess, use the php.ini or the virtual host configuration.

Answer 2

In Linux systems, there can be three separate memory limit settings, one for the Apache DSO module php instance, one for the Command Line php instance and one for the cgi php instance.

Each installation will have its own php.ini file, for example /etc/php5/apache2/php.ini and /etc/php5/cli/php.ini and may have the ability to override the set memory_limit turned off (typically found in shared hosting).

The php.ini file for cli will need to have its memory_limit increased.

Another spot of trouble can occur if you are using the apc cache. It has to be enabled by adding the following to the apc config file.

apc.enable-cli=1

A third source of trouble if you are running under cPanel/WHM is that there can be several different versions of php installed due to the security restrictions put in place. I had to contact our hosting service provider to run that one down. Try using which php from the command line where you are having the failure and make sure it's the same version that's accessible via cron scripts. The php.ini file for that particular version also has to be set.