Benchmark memory usage in PHP

Question

SO,

Specifics

Let us suppose that we have some problem and at least two solutions for it. And what we want to achieve - is to compare effectiveness for them. How to do this? Obviously, the best answer is: do tests. And I doubt there's a better way when it comes to language-specific questions (for example "what is faster for PHP: echo 'foo', 'bar' or echo('foo'.'bar')").

Ok, now we'll assume that if we want to test some code, it's equal to test some function. Why? Because we can wrap that code to function and pass it's context (if any) as it's parameters. Thus, all we need - is to have, for example, some benchmark function which will do all stuff. Here's very simple one:

function benchmark(callable $function, $args=null, $count=1) {    $time = microtime(1);    for($i=0; $i<$count; $i++)    {       $result = is_array($args)?                 call_user_func_array($function, $args):                 call_user_func_array($function);    }    return [       'total_time'   => microtime(1) - $time,       'average_time' => (microtime(1) - $time)/$count,       'count'        => $count    ]; } 

-this will fit our issue and can be used to do comparative benchmarks. Under comparative I mean that we can use function above for code X, then for code Y and, after that, we can say that code X is Z% faster/slower than code Y.

The problem

Ok, so we can easily measure time. But what about memory? Our previous assumption "if we want to test some code, it's equal to test some function" seems to be not true here. Why? Because - it's true from formal point, but if we'll hide code inside function, we'll never be able to measure memory after that. Example:

function foo($x, $y) {    $bar = array_fill(0, $y, str_repeat('bar', $x));    //do stuff }  function baz($n) {    //do stuff, resulting in $x, $y    $bee = foo($x, $y);    //do other stuff } 

-and we want to test baz - i.e. how much memory it will use. By 'how much' I mean 'how much will be maximum memory usage during execution of function'. And it is obvious that we can not act like when we were measuring time of execution - because we know nothing about function outside of it - it's a black box. If fact, we even can't be sure that function will be successfully executed (imagine what will happen if somehow $x and $y inside baz will be assigned as 1E6, for example). Thus, may be it isn't a good idea to wrap our code inside function. But what if code itself contains other functions/methods call?

My approach

My current idea is to create somehow a function, which will measure memory after each input code's line. That means something like this: let we have code

$x = foo(); echo($x); $y = bar(); 

-and after doing some thing, measure function will do:

$memory = memory_get_usage(); $max    = 0;  $x = foo();//line 1 of code $memory = memory_get_usage()-$memory; $max    = $memory>$max:$memory:$max; $memory = memory_get_usage();  echo($x);//second line of code $memory = memory_get_usage()-$memory; $max    = $memory>$max:$memory:$max; $memory = memory_get_usage();  $y = bar();//third line of code $memory = memory_get_usage()-$memory; $max    = $memory>$max:$memory:$max; $memory = memory_get_usage();  //our result is $max 

-but that looks weird and also it does not answer a question - how to measure function memory usage.

Use-case

Use-case for this: in most case, complexity-theory can provide at least big-O estimation for certain code. But:

• First, code can be huge - and I want to avoid it's manual analysis as long as possible. And that is why my current idea is bad: it can be applied, yes, but it will still manual work with code. And, more, to go deeper in code's structure I will need to apply it recursively: for example, after applying it for top-level I've found that some foo() function takes too much memory. What I will do? Yes, go to this foo() function, and.. repeat my analysis within it. And so on.
• Second - as I've mentioned, there are some language-specific things that can be resolved only by doing tests. That is why having some automatic way like for time measurement is my goal.

Also, garbage collection is enabled. I am using PHP 5.5 (I believe this matters)

The question

How can we effectively measure memory usage of certain function? Is it achievable in PHP? May be it's possible with some simple code (like benchmark function for time measuring above)?

Answer 1

After @bwoebi proposed great idea with using ticks, I've done some researching. Now I have my answer with this class:

class Benchmark {    private static $max, $memory;     public static function memoryTick()    {       self::$memory = memory_get_usage() - self::$memory;       self::$max    = self::$memory>self::$max?self::$memory:self::$max;       self::$memory = memory_get_usage();    }     public static function benchmarkMemory(callable $function, $args=null)    {       declare(ticks=1);       self::$memory = memory_get_usage();       self::$max    = 0;        register_tick_function('call_user_func_array', ['Benchmark', 'memoryTick'], []);       $result = is_array($args)?                 call_user_func_array($function, $args):                 call_user_func($function);       unregister_tick_function('call_user_func_array');       return [          'memory' => self::$max       ];    } }  //var_dump(Benchmark::benchmarkMemory('str_repeat', ['test',1E4])); //var_dump(Benchmark::benchmarkMemory('str_repeat', ['test',1E3])); 

-so it does exactly what I want:

• It is a black box
• It measures maximum used memory for passed function
• It is independent from context

Now, some background. In PHP, declaring ticks is possible from inside function and we can use callback for register_tick_function(). So my though was - to use anonymous function which will use local context of my benchmark function. And I've successfully created that. However, I don't want to affect global context and so I want unregister ticks handler with unregister_tick_function(). And that is where troubles are: this function expects string to be passed. So you can not unregister tick handler, which is closure (since it will try to stringify it which will cause fatal error because there's no __toString() method in Closure class in PHP). Why is it so? It's nothing else, but a bug. I hope fix will be done soon.

What are other options? The most easy option that I had in mind was using global variables. But they are weird and also it is side-effect which I want to avoid. I don't want to affect context. But, really, we can wrap all that we need in some class and then invoke tick function via call_user_func_array(). And call_user_func_array is just string, so we can overcome this buggy PHP behavior and do the whole stuff succesfully.

Update: I've implemented measurement tool from this. I've added time measurement and custom callback-defined measurement there. Feel free to use it.

Update: Bug, mentioned in this answer, is now fixed, so there's no need in trick with call_user_func(), registered as tick function. Now closure can be created and used directly.

Update: Due to feature request, I've added composer package for this measurement tool.