How to free memory in Perl?

Question

My code looks like:

 my %var;
 my %var_new={};

 while(1){
     while(my ($k,$v)=each %var){
            &a_sub($v);
     }
     %var={}; # A
     map { $var{$_}=$var_new{$_}; } keys %var_new;
     %var_new={}; # B
 }

 sub a_sub { #....} # will fill %var_new

My program's memory usage goes up and up.

It seems Perl doesn't free memory at Line A and Line B.

How can I manually make Perl to free up memory used %var and %var_new?

Answer 1

If you had used strict and warnings, you would have seen:

Reference found where even-sized list expected

Lines A and B don't do what you think they do. They actually assign as a key the stringified version of a reference to an empty hash, and undef as its value. Even if the hash is now almost empty, memory is not marked as reusable because you didn't use the proper statement for that.

Try using one of the following statements on lines A and B:

undef %var; # this one
%var = ();  # or this one

Answer 2

IIRC, once Perl has allocated memory from the operating system, it holds onto that memory for the life of the process. Where possible, Perl will reuse memory it has already allocated rather than ask the operating system for more, but you won't see the memory used by a process decrease.

Answer 3

I think that using the undef function:

undef %var, %var_new;

might do the trick. That is of course if you don't need what's inside those hashes anymore

Answer 4

It is possible to release memory back to the OS via undef. However there is a difference between $var = undef; and undef($var);

$var = undef; does nothing except resetting flags of the object. Size is still the same

undef($var); does actually delete the pointer and free's the memory of this object - if and only if this is the 'original' variable.

There is an interesting Lightning Talk from Cees Hek - To undef or not to undef

For example the following code goes almost back to its originial memory usage after the call of undef (Tested on Perl 5.30.0 Fedora 31, libc 2.30)

#!perl

use feature 'say';

say "PID: $$";
say "Before Alloc";
<STDIN>;

my $large_string = "A" x 100000000;

say "After Alloc - Before undef";
<STDIN>;

undef($large_string);

say "After undef";
<STDIN>;

But ... this does not really mean that your complex datastructure will free all that memory after just calling undef on it.

The following example illustrates the problem: The main memory consumer is $sneaky. After a call to undef(%hash) the memory usage stays almost the same because $sneaky is still around.

#!perl

use feature 'say';
use Devel::Peek;
$Devel::Peek::pv_limit = 15;

say "PID: $$";
say "BEFORE ALLOC";
<STDIN>;

my $sneaky = "A" x 100000000;
my %hash;
$hash{large_string_inside} = $sneaky;

say "After Alloc - Before undef";
say '$sneaky:';
Dump($sneaky);

say '%hash:';
Dump(%hash);

<STDIN>;

undef(%hash);

say "After undef";

say '$sneaky:';
Dump($sneaky);


say '%hash:';
Dump(%hash);

<STDIN>;

Even walking through the structure and calling undef on each element would not be enough (same goes for nested structures - you can walk them recursively but it won't make a difference).

while(my ($k, $v) = each %hash) {
    undef($v); # THIS IS NOT ENOUGH
}

Why? Because we did not delete the original $sneaky. The only way to give the memory back to the OS would be to call undef($sneaky).

It also does not make a difference if you put a reference of $sneaky into the %hash like this $hash{large_string_inside} = \$sneaky; You would still need to call undef($sneaky) and not undef($hash{large_string_inside})