Need advice for Memory usage in perl

Question

I am not new to perl , but I couldn't able to solve this usecase.

Here is my problem statement:

I am using Activestate perl 5.12 32-bit. I want to hold some 32 x 8MB array of double in Memory at a time. unfortunately, I am getting 'out of memory' error in this case. for example, the code below will lead to out of memory.

my $aref1 = [(.25) x (8*1024*1024)];
my $aref2 = [(.25) x (8*1024*1024)];
my $aref3 = [(.25) x (8*1024*1024)];
my $aref4 = [(.25) x (8*1024*1024)];
my $aref5 = [(.25) x (8*1024*1024)];
my $aref6 = [(.25) x (8*1024*1024)];
my $aref7 = [(.25) x (8*1024*1024)];
my $aref8 = [(.25) x (8*1024*1024)];
my $aref9 = [(.25) x (8*1024*1024)];
my $aref10 = [(.25) x (8*1024*1024)];

Is there any way to efficiently handle it?

Note: In any case, the access to the arrays is needed at any time of execution as fast as possible (since the user cannot wait soo long)

the options I have tried:

1. DBM::Deep - It takes more time
2. PDL - Gives 'out of memory' error for 32 x 8MB array of double

Awaiting your valuable suggestions!

Answer 1

32 arrays * 8 Mi-doubles/array * 8 bytes/double = 2 GiB.

32-bit processes on Windows only have 2GiB of usable address space. Your raw data would take up all available address space, leaving nothing for the data structure's overhead, perl, your program and other variables. The only way it's going to fit in memory is if you switch to a 64-bit Perl. Otherwise, you will have to contend with a necessarily slower solution.