php memory_get_usage(true) vs top %MEM

Question

I have a script, written in PHP that uses the AWS Dynamo PHP API. It runs a long loop where it pulls lots of data from dynamo and then it processes it.

When I watch the process using 'top' I can see the memory usage used by the 'php' process

Inside my script's loop I print the result of memory_get_usage(true)

When I run my test these two value are not even remotely similar...

Should they be? If not why not?

In my test I have a server with 1.7gb of ram, and I have set my php.ini's memory_limit to 64M. I also call gc_enable() at the start of my script, and between each loop call gc_collect_cycles() in the hope of forcing a garbage collection.

When I watch my php script using 'top' I can see the %MEM going up and up, until it eventually gets over 95% and linux kills the php process, which I know from looking at 'dmesg'. When I look at the print outs from each iteration of the loop the memory usage reported by memory_get_usage(true) never gets above 50mb.

Linux thinks the script is using almost 1.7gb, php thinks it's only using 50mb!

What goings on?

Even if the script has memory leaks, I don't understand why memory_get_usage(true) does not account for the memory...

UPDATE

After spending some time commenting out various parts of the processing I am running inside my loop I found that if I remove the following code:

class cMyClass {
    public static function static_cmp_fn(&$a, &$b) {
        if ($a['att'] == $b['att']) { return 0; }
        $ret = ($a['att'] < $b['att']) ? -1 : +1;
        return $ret;
    }
    function DoProcessing(){
        $sort_fn = array("cMyClass", "static_cmp_fn");
        usort($this->m_dictToSort, $sort_fn); 
        unset($sort_fn);
    }

}

php never eats all of the system memory. It seems to me that the usort is leaking memory, I don't know why. What I don't understand is why PHP reports the wrong information about how much memory it is using...

Any ideas?

Answer 1

After spending some time commenting out various parts of the processing I am running inside my loop I found that if I remove the following code:

$sort_fn = array("cMyClass", "static_cmp_fn");
usort($this->m_dictToSort, $sort_fn); 

php never eats all of the system memory. It seems to me that the usort is leaking memory, I don't know why.

Apparently it is. See the manual:

http://php.net/manual/en/function.usort.php

"A couple examples here advocate the use of 'create_function' for sorting, which is tempting to use because of the limitations of usort. But beware this method -- the function created will NOT be freed at the end of the sorting routine, which creates a memory leak. For this reason, this method should probably never be used."

The array() method seems to do something similar. You could declare a wrapper function which calls your method externally, perhaps?

UPDATE

Tried to build a small test case to see what happens. As yet I cannot reproduce the leak; maybe with more data about what static_cmp_fn() does and how m_dictToSort is structured. A simple compare doesn't trigger anything strange. Nor does allocating strings, arrays, or objects inside the loop. The garbage collector kills them off and memory stays low.

I'd further restrict the problem by calling another function which doesn't sort at all, or does a very basic sort, to see whether the problem is in usort doing something funny with its callable, as I thought (it appears it doesn't, and I was wrong) or if something funny is happening inside the compare function.