CLR out of memory exceptions

Question

We are getting a lot of out of memory exceptions and we can't seem to diagnose what is happening. It seems to be an issue that happens that will spike memory usage from 300 MB to over 1 GB in a matter of a few minutes. Now this is an IIS application and has three application domains running in separate thread pools.

We want to record when the memory exceptions are about to happen. So we can attempt to find a pattern. What is the best way to do this?

Is there a way to query the memory usage once a minute to see how high it is and send an alert email. Or maybe write an application that will monitor the CLR's memory usage? An ideas or a direction are more than welcome.

I am using Performance Monitor, but unless I am watching the process it is not that useful. I can only see what is was and when. I have also used the Red Gate Memory Profile tool, which is awesome by the way - only I cannot seem to hit upon the page or process that is causing the exception.

Answer 1

Install ADPlus on your server (it's part of the Windows Debugging Tools). When you start observing unusually high memory usage capture a memory dump using:

adplus -hang -p <PID> -quiet -o <dump file folder path>

<PID> is the process ID of the worker process which you can get from tasklist.exe.

If you're not always around when this issue occurs then you could automate capturing a memory dump using DebugDiag:

1. Use DebugDiag in leak tracking mode to trigger a process dump when either your Private or Virtual memory use reaches a certain threshold. That said, I've not always found this reliable.

2. Use DebugDiag in Crash Mode to capture a dump whenever CLR exceptions are thrown. You can used the Advanced Settings to configure DebugDiag to produce a full memory dump upon encountering a CLR exception of type System.OutOfMemoryException. This is more reliable and will definitely trigger. Only use Advanced Settings -> Exceptions, don't touch the Unconfigured First Chance Exceptions, leave this setting at None.

Once you have your memory dump, fire up WinDBG then load the dump file and load up SOS and start poking about.

Tess Ferrandez's blog is a great .NET debugging resource and she has plenty of articles and labs about how to track down memory leaks (these are archive.org links due to broken links):

If it is broken, fix it you should - memory issue articles

.NET Debugging Demos Lab 6:_Memory Leak
.NET Debugging Demos Lab 6:_Memory Leak - Review
.NET Debugging Demos Lab 7: Memory Leak
.NET Debugging Demos Lab 7: Memory Leak - Review

PerfMon counters are useful and can be used to confirm that you have a memory leak, but they don't tell the whole story. WinDBG and SOS are the tools you need to use to find out where your memory is being used.

Answer 2

Performance Monitor, aka Perfmon, is your friend - it is free and not intrusive and lightweight and can be safely run on production servers if you use less frequent sampling (every few seconds). The minimum it can do is to sample memory and CPU usage for your processes (w3wp.exe) and store them on a file.

Since you have not shared what you are doing in the application I can't suggest performance counters to store, but there are plenty in the ASP.NET and .NET and CLR.

Since you are getting a CLR out of memory exception, my hunch is GC is not working due to a pinned object or something. I doubt it is an unmanaged resource, such as bitmap, not being released although it could be.

Here are a list of counters I suggest:

.NET CLR Memory

• GC Handles
• Pinned Objects

Process for the w3wp.exe running your application

• A couple, but mostly Working Set

ASP.NET

• Managed Memory used