Can I use dtrace on OS X 10.5 to determine which of my perl subs is causing the most memory allocation?

Question

We have a pretty big perl codebase.

Some processes that run for multiple hours (ETL jobs) suddenly started consuming a lot more RAM than usual. Analysis of the changes in the relevant release is a slow and frustrating process. I am hoping to identify the culprit using more automated analysis.

Our live environment is perl 5.14 on Debian squeeze.

I have access to lots of OS X 10.5 machines, though. Dtrace and perl seem to play together nicely on this platform. Seems that using dtrace on linux requires a boot more work. I am hoping that memory allocation patterns will be similar between our live system and a dev OS X system - or at least similar enough to help me find the origin of this new memory use.

This slide deck:

https://dgl.cx/2011/01/dtrace-and-perl

shows how to use dtrace do show number of calls to malloc by perl sub. I am interested in tracking the total amount of memory that perl allocates while executing each sub over the lifetime of a process.

Any ideas on how this can be done?

Answer 1

There's no single way to do this, and doing it on a sub-by-sub basis isn't always the best way to examine memory usage. I'm going to recommend a set of tools that you can use, some work on the program as a whole, others allow you to examine a single section of your code or a single variable.

You might want to consider using Valgrind. There's even a Perl module called Test::Valgrind that will help set up a suppression file for your Perl build, and then check for memory leaks or errors in your script.

There's also Devel::Size which does exactly what you asked for, but on a variable-by-variable basis rather than a sub-by-sub basis.

You can use Devel::Cycle to search for inadvertent circular memory references in complex data structures. While a circular reference doesn't mean that you're wasting memory as you use the object, circular references prevent anything in the chain from being freed until the cycle is broken.

Devel::Leak is a little bit more arcane than the rest, but it basically will allow you to get full information on any SVs that are created and not destroyed between two points in your program's execution. If you check this across a sub call, you'll know any new memory that that subroutine allocated.

You may also want to read the perldebguts section of the Perl manual.

I can't really help more because every codebase is going to wind up being different. Test::Valgrind will work great for some codebases and terribly on others. If you are going to try it, I recommend you use the latest version of Valgrind available and Perl >= 5.10, as Perl 5.8 and Valgrind historically didn't get along too well.

Answer 2

You might want to look at Memory::Usage and Devel::Size

To check the whole process or sub:

use Memory::Usage;
my $mu = Memory::Usage->new();

# Record amount of memory used by current process
$mu->record('starting work');

# Do the thing you want to measure
$object->something_memory_intensive();

# Record amount in use afterwards
$mu->record('after something_memory_intensive()');

# Spit out a report
$mu->dump();

Or to check specific variables:

use Devel::Size qw(size total_size);

my $size = size("A string");

my @foo = (1, 2, 3, 4, 5);
my $other_size = size(\@foo);

my $foo = {
     a => [1, 2, 3],
     b => {a => [1, 3, 4]}
};
my $total_size = total_size($foo);

Answer 3

The answer to the question is 'yes'. Dtrace can be used to analyze memory usage in a perl process.

This snippet of code:

https://github.com/astletron/perl-dtrace-malloc/blob/master/perl-malloc-total-bytes-by-sub.d

tracks how memory use increases between the call and return of every sub in a program. As an added bonus, dtrace seems to sort the output for you (at least on OS X). Cool.

Thanks to all that chimed in. I answered this one myself as the question is really specific to dtrace/perl.