Perl cannot allocate more than 1.1 GB on a Snow leopard Mac server with 32 GB RAM

Question

I have a Mac server (snow leopard) with 32GB RAM. When I try to allocate more than 1.1GB RAM in Perl (v 5.10.0) I get an out of memory error. Here is the script that I used:

#!/usr/bin/env perl

# My snow leopard MAC server runs out of memory at >1.1 billion bytes.  How
# can this be when I have 32 GB of memory installed?  Allocating up to 4
# billion bytes works on a Dell Win7 12GB RAM machine.

# perl -v
# This is perl, v5.10.0 built for darwin-thread-multi-2level
# (with 2 registered patches, see perl -V for more detail)

use strict;
use warnings;

my $s;
print "Trying 1.1 GB...";
$s = "a" x 1100000000;   # ok
print "OK\n\n";

print "Trying 1.2 GB...";
$s = '';
$s = "a" x 1200000000;   # fails
print "..OK\n";

Here is the output that I get:

Trying 1.1 GB...OK

perl(96685) malloc: *** mmap(size=1200001024) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Out of memory!
Trying 1.2 GB...

Any ideas why this is happening?

---

UPDATE 4:42pm 11/14/13

As per Kent Fredric (see 2 posts below), here are my ulimits. Virtual memory defaults to unlimited

$ ulimit -a | grep bytes
data seg size           (kbytes, -d) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
virtual memory          (kbytes, -v) unlimited

$ perl -E 'my $x = "a" x 1200000000; print "ok\n"'
perl(23074) malloc: *** mmap(size=1200001024) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Out of memory!

$ perl -E 'my $x = "a" x 1100000000; print "ok\n"'
ok

I tried setting virtual memory to 10 billion but to no avail.

$ ulimit -v 10000000000   # 10 billion

$ perl -E 'my $x = "a" x 1200000000; print "ok\n"'
perl(24275) malloc: *** mmap(size=1200001024) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Out of memory!

Answer 1

You're using a 32-bit build of Perl (as shown by perl -V:ptrsize), but you need a 64-bit build. I recommend installing a local perl using perlbrew.

This can be achieved by passing -Duse64bitall to Configure when installing Perl.

This can be achieved by passing --64all to perlbrew install when installing Perl.

(For some odd reason, perl -V:use64bitall says this was done, but it clearly wasn't.)

Answer 2

Seems like this could be related to the problem. This only really is worthy of a comment, but its too complex to put as one without it being entirely illegible

perlbrew exec --with=5.10.0 memusage perl -e '$x = q[a] x 1_000_000_000; print length($x)'
5.10.0
==========
1000000000
Memory usage summary: heap total: 2000150514, heap peak: 2000141265, stack peak: 4896

Yes, thats 2 G of memory for 1 G of text.

Now with 2G ...

perlbrew exec --with=5.10.0 memusage perl -e '$x = q[a] x 1_000_000_000; $y = q[a] x 1_000_000_000; print length($x)+length($y)'
5.10.0
==========
2000000000
Memory usage summary: heap total: 4000151605, heap peak: 4000142092, stack peak: 4896

Yikes. That would certainly hit the 32Bit limit if you had one.

I was spoiled and doing my testing on 5.19.5, which has a notable improvement, namedly copy-on-write strings, which greatly reduces memory consumption:

perlbrew exec --with=5.19.5 memusage perl -e '$x = q[a] x 1_000_000_000; $y = q[a] x 1_000_000_000; print length($x)+length($y)'
5.19.5
==========
2000000000
Memory usage summary: heap total: 2000157713, heap peak: 2000150396, stack peak: 5392

So either way, if you're using any version of Perl at all other than a development one, you need to expect it to eat twice the memory you need.

If there's a memory limit for some reason around the 2G window for 32bit processes, then you will hit that with a 1G string.

Why does Copy On Write matter?

Well, when you do

$a = $b

$a is a copy of $b

So when you do

$a = "a" x 1_000_000_000

First, it expands the right hand side, creating a variable, and then makes a copy to store in $a.

You can prove this by eliminating the copy as follows:

perlbrew exec --with=5.10.0 memusage perl -e 'print length(q[a] x 1_000_000_000)'
5.10.0
==========
1000000000
Memory usage summary: heap total: 1000150047, heap peak: 1000140886, stack peak: 4896

See, all I did was removed the intermediate variable, and the memory usage halved!

:S

Though because 5.19.5 only makes references to the original string, and copies it when written to, it is efficient by default, so removal of the intermediate variable has negligible benefits

perlbrew exec --with=5.19.5 memusage perl -e 'print length(q[a] x 1_000_000_000)'
5.19.5
==========
1000000000
Memory usage summary: heap total: 1000154123, heap peak: 1000145146, stack peak: 5392