How do I debug a memory dump of a spiking ASP.NET process?

Question

Sorry, I couldn't figure out a good way to phrase my real question.

I run a high-traffic ASP.NET site on a 64-bit machine. I have IIS running in 32-bit mode, however, due to some legacy components of the app. I am running this particular web app inside an application pool that has the web garden option on (running 6 processes inside an 8 core machine).

Once or twice a week one of the processes will skyrocket into 100% CPU utilization, causing a giant slowdown for the site, so my plan was to wait until that happens, memory dump the offending process, then poke around WinDbg to zero in on the thread that's spiking to see where the code is spinning its wheels.

I've debugged using WinDbg before to figure out what was causing a deadlock on the site, but that was several months ago and I can't remember how I got it working. (As a side note, this is a lesson to document everything you do.)

I'm running WinDbg on the Windows 2003 server that's running the site, so as to prevent any DLL version problems. Here have been my steps so far, please let me know where I'm going wrong to get the error message that I'm getting.

1. I first memory dump the spiking process using UserDump, with the following command, where 3389 is the ID of the process:

   userdump -k 3389

2. I load the dump into the x86 edition of WinDbg.

3. Since I'm running 32-bit on a 64-bit machine, I first load the memory dump and then:

   .load wow64exts

   .effmach x86

4. I make sure that my symbol path includes the directory that contains my apps PDB files:

   .sympath+ c:\inetpub\myapp\bin

5. Running just `.load SOS' fails with an error of "The system cannot find the file specified", so I go the fully qualified route of the following, which works:

   .load c:\windows\microsoft.net\framework\v2.0.50727\sos

From here, I'm lost. I try any of the SOS commands, like !threads, only to get this error:

Failed to load data access DLL, 0x80004005

That error is also accompanied by the numbered list of items that I should be verifying. I have verified that I am running the most current version of the debugger, mscordacwks.dll is in fact in the same directory as the mscorwks.dll file, and I'm debugging on the same architecture as the dump file.

I've also run the magical ".cordll -ve -u -l" command, but that doesn't solve anything. I'm always greeted with "CLR DLL status: No load attempts" when I execute that. Then I try ".reload", which yields a handful of warnings like "WARNING: wldap32 overlaps dnsapi". I wish it said something like "CLRDLL: Loaded DLL C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll". But it doesn't.

Answer 1

Try executing !sw before running the sos commands. See this blog post.