json_decode and memory clear

Question

I have a list of large JSON files (smallest files are 500 Ko and biggest are 100 Mo).

I need to process each file independently. My problem is that the memory usage is growing more and more after each file, even if I clear all the memory.

Example:

foreach ($files as $file) {
    json_decode(file_get_contents($file->getRealpath()), true);

    $memory = memory_get_usage(true);
    echo 'Memory: '.@round($memory / pow(1024,($i=floor(log($memory, 1024)))), 2).' '.['b', 'kb', 'mb', 'gb', 'tb', 'pb'][$i]."\n";

    gc_collect_cycles();
}

Result:

Memory: 6 mb
(...)
Memory: 6 mb
Memory: 6 mb
Memory: 10 mb
Memory: 10 mb
Memory: 10 mb
(...)
Memory: 12 mb
Memory: 12 mb
Memory: 12 mb
(...)
Memory: 490 mb
Memory: 490 mb
Memory: 490 mb
(...)
Memory: 946 mb
Memory: 944 mb
Memory: 944 mb
(...)

Memory is growing more and more until PHP tells me he can't get more. As you can see I'm doing nothing in this example except a json_decode(), no variable assigned or whatever else. So why is my memory growing like this, and how can I clear it?

Answer 1

Check the size of the file you are trying to get contents. That could be larger so it costs memory usage

Or

You need to check the which variable holds excessive memory, you can use strlen() it does not give you exact memory held by var but length is helpful to find the approximate value.

And you should unset the unused variables to clear the memory.

unset($decoded_data);

Or set

$var = null

When you are using unset, the memory will only be freed whenever garbage collector decides, but when you are setting a variable to a different value (null in this case), then you might get some memory freed of course with the cost of CPU.

I will recommend you to use

https://github.com/salsify/jsonstreamingparser

This is a simple, streaming parser for processing large JSON documents. Use it for parsing very large JSON documents to avoid loading the entire thing into memory, which is how just about every other JSON parser for PHP works.