Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Need an array-like structure in PHP with minimal memory usage

In my PHP script I need to create an array of >600k integers. Unfortunately my webservers memory_limit is set to 32M so when initializing the array the script aborts with message

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 71 bytes) in /home/www/myaccount/html/mem_test.php on line 8

I am aware of the fact, that PHP does not store the array values as plain integers, but rather as zvalues which are much bigger than the plain integer value (8 bytes on my 64-bit system). I wrote a small script to estimate how much memory each array entry uses and it turns out, that it's pretty exactly 128 bytes. 128!!! I'd need >73M just to store the array. Unfortunately the webserver is not under my control so I cannot increase the memory_limit.

My question is, is there any possibility in PHP to create an array-like structure that uses less memory. I don't need this structure to be associative (plain index-access is sufficient). It also does not need to have dynamic resizing - I know exactly how big the array will be. Also, all elements would be of the same type. Just like a good old C-array.


Edit: So deceze's solution works out-of-the-box with 32-bit integers. But even if you're on a 64-bit system, pack() does not seem to support 64-bit integers. In order to use 64-bit integers in my array I applied some bit-manipulation. Perhaps the below snippets will be of help for someone:

function push_back(&$storage, $value) {     // split the 64-bit value into two 32-bit chunks, then pass these to pack().     $storage .= pack('ll', ($value>>32), $value); }  function get(&$storage, $idx) {     // read two 32-bit chunks from $storage and glue them back together.     return (current(unpack('l', substr($storage, $idx * 8, 4)))<<32 |             current(unpack('l', substr($storage, $idx * 8+4, 4)))); } 
like image 835
Alexander Tobias Bockstaller Avatar asked Jan 24 '14 13:01

Alexander Tobias Bockstaller


People also ask

Do arrays use a lot of memory?

The reason that cell arrays take up so much more memory is because they require a lot more overhead to store metadata about each element in the cell array, since each element could have a different type, size, etc.

What are the 3 types of PHP arrays?

Create an Array in PHP In PHP, there are three types of arrays: Indexed arrays - Arrays with a numeric index. Associative arrays - Arrays with named keys. Multidimensional arrays - Arrays containing one or more arrays.

What is difference between array and [] in PHP?

Note: Only the difference in using [] or array() is with the version of PHP you are using. In PHP 5.4 you can also use the short array syntax, which replaces array() with [].

How much memory does each PHP process consume?

When I set the PHP memory limit to e.g. 128 MB for each of these daemons, the processes will only get killed once they reach 128 MB according to PHP's own measurements. However, according to ps , the processes will be using around 200 MB each by that time.


1 Answers

The most memory efficient you'll get is probably by storing everything in a string, packed in binary, and use manual indexing to it.

$storage = '';  $storage .= pack('l', 42);  // ...  // get 10th entry $int = current(unpack('l', substr($storage, 9 * 4, 4))); 

This can be feasible if the "array" initialisation can be done in one fell swoop and you're just reading from the structure. If you need a lot of appending to the string, this becomes extremely inefficient. Even this can be done using a resource handle though:

$storage = fopen('php://memory', 'r+'); fwrite($storage, pack('l', 42)); ... 

This is very efficient. You can then read this buffer back into a variable and use it as string, or you can continue to work with the resource and fseek.

like image 101
deceze Avatar answered Oct 12 '22 15:10

deceze