Profiling memory usage in Mathematica

Question

Is there any way to profile the mathkernel memory usage (down to individual variables) other than paying $$$ for their Eclipse plugin (mathematica workbench, iirc)?

Right now I finish execution of a program that takes multiple GB's of ram, but the only things that are stored should be ~50MB of data at most, yet mathkernel.exe tends to hold onto ~1.5GB (basically, as much as Windows will give it). Is there any better way to get around this, other than saving the data I need and quitting the kernel every time?

EDIT: I've just learned of the ByteCount function (which shows some disturbing results on basic datatypes, but that's besides the point), but even the sum over all my variables is nowhere near the amount taken by mathkernel. What gives?

Answer 1

One thing a lot of users don't realize is that it takes memory to store all your inputs and outputs in the In and Out symbols, regardless of whether or not you assign an output to a variable. Out is also aliased as %, where % is the previous output, %% is the second-to-last, etc. %123 is equivalent to Out[123].

If you don't have a habit of using %, or only use it to a few levels deep, set $HistoryLength to 0 or a small positive integer, to keep only the last few (or no) outputs around in Out.

You might also want to look at the functions MaxMemoryUsed and MemoryInUse.

Of course, the $HistoryLength issue may or not be your problem, but you haven't shared what your actual evaluation is. If you're able to post it, perhaps someone will be able to shed more light on why it's so memory-intensive.

Answer 2

Here is my solution for profiling of memory usage:

myByteCount[symbolName_String] := 
  Replace[ToHeldExpression[symbolName], 
   Hold[x__] :> 
    If[MemberQ[Attributes[x], Protected | ReadProtected], 
     Sequence @@ {}, {ByteCount[
       Through[{OwnValues, DownValues, UpValues, SubValues, 
          DefaultValues, FormatValues, NValues}[Unevaluated@x, 
         Sort -> False]]], symbolName}]];

With[{listing = myByteCount /@ Names[]},
 Labeled[Grid[Reverse@Take[Sort[listing], -100], Frame -> True, 
   Alignment -> Left], 
  Column[{Style[
     "ByteCount for symbols without attributes Protected and \
ReadProtected in all contexts", 16, FontFamily -> "Times"], 
    Style[Row@{"Total: ", Total[listing[[All, 1]]], " bytes for ", 
       Length[listing], " symbols"}, Bold]}, Center, 1.5], Top]]

Evaluation the above gives the following table: