Is There a .Net Memory Profiler that will track all allocations on the Large Object Heap? [closed]

Question

Most .NET memory profilers that I've tried allow you to take snapshots of memory.

However, I'm trying to diagnose an issue where I'm ending up with huge amounts of memory allocated to .NET that is indicated as "free" by the ANTS profiler. (I've confirmed this problem with other profilers like Mem Profiler and the CLR profiler.

ANTS is showing that I have a large amount of memory fragmentation (100% of free memory with 150MB as the largest chunk.) The total size of all the objects in the heap is 180MB. I have 553 MB allocated to .NET and 152 allocated to "Unmanaged".

However, the size of the Large Object Heap (LOH) is only 175kb. This is the actual size of the objects allocated there. I don't allocate any objects that end up on the LOH permanently.

Hence, my problem, some where along the line, I suspect I am somehow allocating large objects (over the 85k limit for the LOH) and then disposing them.

I'm reading large amounts of data (estimating here at several MB) from databases (Oracle, Sql Server), copying this data to object arrays in memory, and processing the data into indexes (arrays, dictionaries, etc) for easier searching/filtering/processing.

My guess is, the data reader(s) are allocating a lot of space temporarily. However, I don't have a good way to pause the program and take a snapshot of the memory.

What I'd like is a profiler that keeps track of every object allocated on the LOH so I can figure out what is causing the LOH fragmentation and excessive memory usage (the memory is not returned to the OS, so it looks like my process is taking 1GB of memory to store 200MB of allocated objects.) I'm guess the memory is not returned because the LOH is not compacted, so I'm stuck with all this memory for the life of my process, which can be weeks (it runs as a windows service.)

Edit: My problem is that my .NET application is using a lot of memory that I can't trace.

Edit1: I've used the Visual Studio memory profiler. While it does tell me all objects that are instantiated, how many, total bytes, etc, I doesn't give me a hint as to why I end up with so much free memory. My only hint/clue is what ANTS is telling me: "Memory Fragmentation is restricting the size of the objects that can be allocated." and I have a lot of unused memory allocated to .NET that is not used.

Edit2: More profiling shows that I have some short lived large objects allocated on the LOH. However, the total amount allocated on the LOH is never more the 3 to 4 MB. However, during this time the private bytes shoot through the roof, doubling and tripling, while the size of my actually allocated objects (on all heaps) only grows slightly. For instance bytes in all heaps is 115MB but my private bytes are over 512 MB.

ANTS is telling me clearly that I am having a problem with memory fragmentation. Turns out I am creating short lived objects on the LOH. However, these objects never total more than 3 or 4 MB. So these short lived large objects (appear to?) are fragmenting the heck out of the LOH.

To respond to Eric Lippert, and the Disney Land parking lot analogy (which is great).

It's like someone parks in a spot for a few minutes, and then leaves. That spot is then reserved (no one else can park there) until I repave the parking lot!

I first starting investigating this when Visual Studio warned me of memory usage and advised switching to x64. (I forget the warning number, quick google doesn't find it). So switching to x64 alleviates the immediate problem, but doesn't address the underlying problem.

It's like I have a parking lot for 1000 cars, but after I put 100 cars in it, my parking attendants are screaming that it's full...

Luckily I have a huge VMware cluster at my disposal and an understanding admin. I've been allocated 8 cpu's and 8 GB of memory. So as far as a problem, I can deal with it, I just throw resources at it. Also, (as i said above) I switched to x64 a while back as Visual Studio kept nagging me with a warning about However, I'd like to figure out what it allocated on the LOH to see if I an mitigate this Heap fragmentation with some small code changes. Perhaps a fool's errand, given that I can just throw resources at it.

The application runs fine, it's fast with the occasional GC pause. But mostly I can live with the situation, I'd just like to know what objects are causing it. My suspicions are some short lived dictionary's that I haven't tracked down yet.

Edit3: http://msdn.microsoft.com/en-us/magazine/cc188781.aspx

ObjectAllocatedByClass does not track allocations of the large object heap, but ObjectAllocated does. By comparing the notifications from the two, an enterprising soul should be able to figure out what is in the large object heap as opposed to the normal managed heap.

So it looks like this can be done. However, my C++ skills are way to rusty to dig into this (maybe sometime in the future if I get more time). I was hoping that a profiler would provide this out of box.

Answer 1

After experimenting, I have been able to be notified when the GC is removing things from a generation, but not when its putting it in there.

Since LOH is not generation specific, and there is no specific event I can access for notification of LOH insertions, then the only alternative I can offer is to debug the application, take a crash dump - OR - run it locally and use WinDBG. Here's how you can do it:

1. Download the Windows SDK 7
2. Copy the %Microsoft_NET_Framework%\sos.dll to the WinDBG directory
3. IN WinDBG Click File -> Open Executable -> point to your executable
4. In the command bar at the bottom type g (Go)
5. Monitor the memory and when you want to analyse it, go to WinDBG -> Debug menu -> Break
6. Type load .sos - to load the .NET extensions
7. Type !dumpheap -min 85000 - this will list large objects, which should be residing on the LOH

         Address               MT     Size
0000000012a17048 000007fee7ae6ae8   400032     
0000000012a78b00 000007fee7ae6ae8   400032     
0000000012ada5b8 000007fee7ae6ae8   400032     
0000000012b3c070 000007fee7ae6ae8   400032     
0000000012b9db28 000007fee7ae6ae8   400032

Next we need to go through each of these and find out whats in them.

1. Copy the first column (the object address) into the clipboard
2. Type !do <paste from clipboard>
3. This will list the contents of the object, its type and size.

CLR Version: 4.0.30319.261
SOS Version: 4.0.30319.239
Name:        System.String
MethodTable: 000007fee7ae6ae8
EEClass:     000007fee766ed68
Size:        400026(0x61a9a) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      8470737076787475867884758166807183888774746571677189..
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee7aec9d0  4000103        8         System.Int32  1 instance           200000 m_stringLength
000007fee7aeb510  4000104        c          System.Char  1 instance               38 m_firstChar
000007fee7ae6ae8  4000105       10        System.String  0   shared           static Empty
                                 >> Domain:Value  000000000055fe50:0000000002a11420 <<

And the lines you want to look for are:

Size:        400026(0x61a9a) bytes
String:      8470737076787475867884758166807183888774746571677189..

1. Do this for each object

(However, I am assuming it would be a string, so check out the 'Name' property just to be sure. It could be an array.)