Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Finding a Perl memory leak

SOLVED see Edit 2

Hello,

I've been writing a Perl program to handle automatic upgrading of local (proprietary) programs (for the company I work for).

Basically, it runs via cron, and unfortunately has a memory leak (or something similar). The problem is that the leak only happens when I'm not looking (aka when run via cron, not via command line).

My code does not contain any circular (or other) references, so the commonly cited tools will not help me (Devel::Cycle, Devel::Peek).

How would I go about figuring out what is using so much memory that the kernel kills it?

Basically, the code SFTPs into a server (using ```sftp...`` `), calls OpenSSL to verify the file, and then SFTPs more if more files are needed, and installs them (untars them).

I have seen delays (~15 sec) before the first SFTP session, but it has never used so much memory as to be killed (in my presence).

If I can't sort this out, I'll need to re-write in a different language, and that will take precious time.

Edit: The following message is printed out by the kernel which led me to believe it was a memory leak:

[100023.123] Out of memory: kill process 9568 (update.pl) score 325406 or a child
[100023.123] Killed Process 9568 (update.pl)

I don't believe it is an issue with cron because of the stalling (for ~15 sec, sometimes) when running it via the command-line. Also, there are no environmental variables used (at least by what I've written, maybe underlying things do?)

Edit 2: I found the issue myself, with help from the below comment by mobrule (in response to this question). It turns out that the script was called from a crontab of a user (non-root) just once a day and that (non-root privs) caused a special infinite loop situation.

Sorry guys, I feel kinda stupid for not finding this before, but thanks.

mobrule, if you submit your comment as an answer, I will accept it as it lead to me finding the problem.

End Edits

Thanks, Brian

P.S. I may be able to post small snippets of code, but not the whole thing due to company policy.

like image 810
HalfBrian Avatar asked Nov 02 '09 20:11

HalfBrian


People also ask

How do you detect a memory leak?

The best approach to checking for the existence of a memory leak in your application is by looking at your RAM usage and investigating the total amount of memory been used versus the total amount available. Evidently, it is advisable to obtain snapshots of your memory's heap dump while in a production environment.

How do I find a memory leak in Delphi?

All that it takes to start working with it is add a single line in your project: ReportMemoryLeaksOnShutdown := True; And voilà, your application will report all memory leaks when it shuts down. If there are leaks at the end of your application, a dialog will be displayed showing all the leaks.


2 Answers

You could try using Devel::Size to profile some of your objects. e.g. in the main:: scope (the .pl file itself), do something like this:

use Devel::Size qw(total_size);

foreach my $varname (qw(varname1 varname2 ))
{
    print "size used for variable $varname: " . total_size($$varname) . "\n";
}

Compare the actual size used to what you think is a reasonable value for each object. Something suspicious might pop out immediately (e.g. a cache that is massively bloated beyond anything that sounds reasonable).

Other things to try:

  • Eliminate bits of functionality one at a time to see if suddenly things get a lot better; I'd start with the use of any external libraries
  • Is the bad behaviour localized to just one particular machine, or one particular operating system? Move the program to other systems to see how its behaviour changes.
  • (In a separate installation) try upgrading to the latest Perl (5.10.1), and also upgrade all your CPAN modules
like image 156
Ether Avatar answered Oct 10 '22 23:10

Ether


How do you know that it's a memory leak? I can think of many other reasons why the OS would kill a program.

The first question I would ask is "Does this program always work correctly from the command line?". If the answer is "No" then I'd fix these issues first.

On the other hand if the answer is "Yes", I would investigate all the differences between having the program executed under cron and from the command line to find out why it is misbehaving.

like image 22
David Harris Avatar answered Oct 10 '22 22:10

David Harris