My (huge) application throws an OutOfMemoryException, now what?

Question

This is by far the most complex software I've built and now it seems to be running out of memory at some point. I haven't done extensive testing yet, because I'm a bit lost how I should approach the problem at hand.

HandleCount: 277
NonpagedSystemMemorySize: 48136
PagedMemorySize: 1898590208
PagedSystemMemorySize: 189036
PeakPagedMemorySize: 1938321408
VirtualMemorySize: 2016473088
PeakVirtualMemory: 2053062656
WorkingSet: 177774592
PeakWorkingSet: 883834880
PrivateMemorySize: 1898590208
PriviligedProcessorTime: 00:00:15.8593750
UserProcessorTime: 00:00:01.6562500
TotalProcessorTime: 00:00:17.5156250
GDI Objects: 30
User Objects: 27

I have an automated global exception catcher that upon exception gathers the above information (using System.Diagnostics.Process) - along with the exception information, log and a screen shot - and e-mails me everything.

This has been working nicely as I've been able to plug bugs based on the e-mailed information. This is, up until now. The software is tens of thousands of lines and uses managed and unmanaged resources.

I could start going through the code, line by line, but some how I get a feeling this might not be the best approach to try to deduce the memory build-up problem.

As I've never done this kind of analysis before, how would you suggest to approach this kind of a problem?

Answer 1

We provide a tool for that.

http://msdn.microsoft.com/en-us/library/ms979205.aspx

CLR Profiler enables you to look at the managed heap of a process and investigate the behavior of the garbage collector. Using the various views in the tool, you can obtain useful information about the execution, allocation, and memory consumption of your application.

Using CLR Profiler, you can identify code that allocates too much memory, causes too many garbage collections, and holds on to memory for too long.

Answer 2

There are a couple of options. Dedicated memory profilers such as ANTS Memory Profiler from RedGate can be very useful for troubleshooting this kind of problem.

If you don't want to spend money on a dedicated tool, you can also use WinDbg (part of Debugging tools for Windows, a free download from Microsoft). It can show you heap usage for the managed heap, the various AppDomain heaps and so forth.

Have a look at this blog for hints on using WinDbg.

Keep in mind that troubleshooting out of memory can be hard, as you usually don't see the actual problem but merely a symptom. So unlike a crash where the call stack will give you a pretty good indication of the source of the problem, the call stacks for a process with OOM may reveal very little.

In my experience you have to look at where memory is used. It could be on the managed heap, in which case you have to find out if something is holding on to instances longer than necessary. However, it could also be related to loading lots of assemblies (typically assemblies generated on the fly).