Problems with PHP when trying to create big array

Question

Here is my code, which creates 2d array filled with zeros, array dimensions are (795,6942):

function zeros($rowCount, $colCount){
    $matrix = array();
    for ($rowIndx=0; $rowIndx<$rowCount; $rowIndx++){
        $matrix[] = array();
        for($colIndx=0; $colIndx<$colCount; $colIndx++){
            $matrix[$rowIndx][$colIndx]=0;
        }
    }
    return $matrix;
}

$matrix = zeros(795,6942);

And here is the error that I receive:

Allowed memory size of 134217728 bytes exhausted (tried to allocate 35 bytes)

Any ideas how to solve this?

Answer 1

As a quick calculation, you are trying to create an array that contains :

795*6942 = 5,518,890

integers.

If we consider that one integer is stored on 4 bytes (i.e. 32 bits ; using PHP, it not be less), it means :

5518890*4 = 22,075,560

bytes.

OK, quick calculation... result is "it should be OK".

But things are not that easy, unfortunatly :-(
I suppose it's related to the fact that data is stored by PHP using an internal data-structure that's much more complicated than a plain 32 bits integer

Now, just to be curious, let's modify your function so it outputs how much memory is used at the end of each one of the outer for-loop :

function zeros($rowCount, $colCount){
    $matrix = array();
    for ($rowIndx=0; $rowIndx<$rowCount; $rowIndx++){
        $matrix[] = array();
        for($colIndx=0; $colIndx<$colCount; $colIndx++){
            $matrix[$rowIndx][$colIndx]=0;
        }
        var_dump(memory_get_usage());
    }
    return $matrix;
}

With this, I'm getting this kind of output (PHP 5.3.2-dev on a 64bits system ; memory_limit is set to 128MB -- which is already a lot !) :

int 1631968
int 2641888
int 3651808
...
...
int 132924168
int 133934088
Fatal error: Allowed memory size of 134217728 bytes exhausted

Which means each iteration of the outer for-loop requires something like 1.5 MB of memory -- and I only get to 131 iterations before the script runs out of memory ; and not 765 like you wanted.

Considering you set your memory_limit to 128M, you'd have to set it to something really much higher -- like

128*(765/131) = 747 MB

Well, even with

ini_set('memory_limit', '750M');

it's still not enough... with 800MB, it seems enough ;-)

But I would definitly not recommend setting memory_limit to such a high value !

(If you have 2GB of RAM, your server will not be able to handle more than 2 concurrent users ^^ ;; I wouldn't actually test this if my computer had 2GB of RAM, to be honest)

The only solution I see here is for you to re-think your design : there has to be something else you can do than use this portion of code :-)

(BTW : maybe "re-think your design" means using another language PHP : PHP is great when it comes to developping web-sites, but is not suited to every kind of problem)

Answer 2

The default PHP array implementation is very memory-intensive. If you are just storing integers (and lots of them), you probably want to look at SplFixedArray. It uses a regular contiguous memory block to store the data, as opposed to the traditional hash structure.