Help catching StackOverflowException with WinDbg and ADPlus

Question

Short Version

I want an ADPlus script that will do a full memory dump on the first-chance StackOverflowException, before anything is cleaned up, and ignore all other exception types.

Log Version

After a release of new ASP.NET code, we started getting intermittent StackOverflowExceptions. We've looked for infinite recursions and all the usual suspects in the revisions added since the last known good install, and can't find anything. The website will run for up to an hour, and then crash down.

We've used WinDbg and SOS and attempted to get crash logs using ADPlus, using this command:

adplus -crash -o D:\Crash -NoDumpOnFirst -iis

The reason for -NoDumpOnFirst is that we can only reproduce this error in production on busy servers. In order to do a minidump on each first-chance exception (hey, it happens) the debugger has to pause the IIS worker process long enough to write out a 16 meg file, so requests queue up and the application becomes unstable. Because the error can take up to an hour to rear it's ugly head, this is problematic.

So with -NoDumpOnFirst, I get a dump file that WinDbg outputs these threads for:

PDB symbol for mscorwks.dll not loaded
ThreadCount: 69
UnstartedThread: 0
BackgroundThread: 69
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
                                      PreEmptive   GC Alloc           Lock
       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
XXXX    1  c6c 000fa758  11808221 Disabled 3b49ee4c:3b49efe8 00120888     1 Ukn (Threadpool Worker)
XXXX    2 1294 000fd258      b220 Enabled  00000000:00000000 000df4e0     0 Ukn (Finalizer)
XXXX    3 1eb0 0011cdd0    80a220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Completion Port)
XXXX    4 1b3c 00120198      1220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX    5 1280 00138118   880a220 Enabled  2633de9c:2633ee08 000df4e0     0 Ukn (Threadpool Completion Port)
XXXX    6 1db8 00158a48  1180a221 Disabled 4b5a7e2c:4b5a82e8 00120888     1 Ukn (Threadpool Worker)
XXXX    9 141c 00162008   180a220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Worker)
XXXX    7 1574 00174008   180a220 Enabled  4d46b6a8:4d46c158 00120888     2 Ukn (Threadpool Worker)
XXXX    c 16c8 0016b7a8   180a220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Worker)
XXXX    8 1384 00162878   180a220 Enabled  284e26a4:284e45d8 000df4e0     0 Ukn (Threadpool Worker)
XXXX    b 1c10 0016b3d8   180a220 Enabled  3ed2dae0:3ed2dfe8 00120888     2 Ukn (Threadpool Worker)
XXXX    a 1814 0016b008   180a220 Disabled 28816384:28816638 00120888     1 Ukn (Threadpool Worker)
XXXX    d  1fc 1b4d1ff0       220 Enabled  319f89a4:319fa41c 000df4e0     0 Ukn
XXXX    e 1864 1b4e3d20   180b220 Enabled  4b2c5be0:4b2c6150 000df4e0     0 Ukn (Threadpool Worker)
XXXX    f 13bc 1b57caf8   200b220 Enabled  4cc71584:4cc73414 00120888     1 Ukn
XXXX   10  72c 1f5124a8   180b220 Enabled  3b4b3414:3b4b4fe8 00120888     2 Ukn (Threadpool Worker)
XXXX   11 1fd0 1f526398   180b220 Disabled 4d46f41c:4d470158 00120888     1 Ukn (Threadpool Worker)
XXXX   12 1f10 1f52f1c8   180b220 Enabled  28812c14:28814638 00120888     2 Ukn (Threadpool Worker)
XXXX   13 1b84 1f53a420       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   14 18a4 1f570978   180b220 Enabled  263e18b4:263e2e28 000df4e0     0 Ukn (Threadpool Worker)
XXXX   15 1a98 1f57f0a0   180b220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Worker)
XXXX   16  1b4 1f583628   180b220 Enabled  495781ec:4957914c 00120888     2 Ukn (Threadpool Worker)
XXXX   17  b90 1f585dc8   180b220 Enabled  265cbe48:265ccba4 000df4e0     0 Ukn (Threadpool Worker)
XXXX   18 1590 1f613c60       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   19 1850 1f5fad90       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   1a  c78 1f60d3f0       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   1c 1bd8 2121f1b0       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   1d  494 1b4a8c10       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   1e  898 2120f120       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   1f 1820 21355ff8       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   20 15b0 3570e120       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   21 18b0 359ca008       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   22  75c 35a58948       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   25 1a18 213ac8f8   880b220 Disabled 3219a830:3219b450 00120888     1 Ukn (Threadpool Completion Port) System.StackOverflowException (0e3200a4)
XXXX   29 1b74 3598e620   180b220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Worker)
XXXX   2a  9b8 3598dbe0   180b220 Enabled  2880ef2c:28810638 000df4e0     0 Ukn (Threadpool Worker)
XXXX   2b 1eac 1f6f6288   180b220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Worker)
XXXX   2d  2f4 211759e8   180b220 Disabled 2634eacc:2634ee08 00120888     1 Ukn (Threadpool Worker)
XXXX   2e 1e3c 35c2eb60   880b220 Enabled  4b5a5758:4b5a62e8 000df4e0     0 Ukn (Threadpool Completion Port)
XXXX   30  394 35c394f8   180b220 Enabled  4cef7930:4cef90d4 000df4e0     0 Ukn (Threadpool Worker)
XXXX   31 1e64 35c39128   180b220 Disabled 288110b0:28812638 00120888     1 Ukn (Threadpool Worker)
XXXX   32 1af8 35a58578   180b220 Enabled  3b48e7cc:3b48efe8 000df4e0     0 Ukn (Threadpool Worker)
XXXX   34 1d44 1f6a6c88   180b220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Worker)
XXXX   35 197c 212088e0   180b220 Enabled  49389ba8:4938af40 000df4e0     0 Ukn (Threadpool Worker)
XXXX   36 1e2c 35c1d980       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   38 1ddc 212d03d8       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   39  288 212d0008       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   3a 1694 212bf958       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   3b  be4 212ccc40       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   37  ccc 35c4d6d0       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   3c 14ec 35c55af0       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   41 1d94 35c38c08   180b220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Worker)
XXXX   24  130 35746a50   180b220 Enabled  2670ae48:2670cc00 000df4e0     0 Ukn (Threadpool Worker)
XXXX   2f 1404 35c1d350   180b220 Enabled  00000000:00000000 000df4e0     0 Ukn (Threadpool Worker)
XXXX   43 1ae8 35c25cb8   180b220 Disabled 3b4c28e0:3b4c2fe8 00120888     1 Ukn (Threadpool Worker)
XXXX   44 18ac 212cc870   180b220 Disabled 4957e728:4957f14c 00120888     1 Ukn (Threadpool Worker)
XXXX   45 18b4 212bf588   180b220 Disabled 3b4c05dc:3b4c0fe8 00120888     1 Ukn (Threadpool Worker)
XXXX   46 1c0c 21239858       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   47  4fc 21188b68       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   48 1198 35caa2a8       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   49 1f9c 21147af8       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   4a 1adc 35cc6908       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   4b  ce8 35c60e30       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   4d  6f0 35d05aa0       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   4e 1ee8 35c1b6b0       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   42 1d7c 35d9a230       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   3d  7d8 212e1b28       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   23  c0c 503ea010       220 Enabled  00000000:00000000 000df4e0     0 Ukn
XXXX   27 1f44 503cdf08       220 Enabled  00000000:00000000 000df4e0     0 Ukn

Trying to print the exception shows that there is no stack trace, and other methods complain that it's unmanaged code. My guess is that as the dump is created at process death, all the threads have been garbage collected and there's no information left to get.

I would really like to have the debugger perform a full dump on the first-chance of the StackOverflowException and ignore all other exception types. I know that ADPlus can use a config file - http://msdn.microsoft.com/en-us/library/cc409304.aspx - but the format is all greek to me. Can anyone show me how to make an ADPlus script that will do this?

...of course if you look at the thread list above and you know exactly what's wrong, or could figure it out if I gave you any more information, you could just tell me that too.

Resolution Attempt 1

Thank you deemok for the answer below, it wasn't quite right but that pushed me in the right direction. The exception code for Stack Overflow was incorrect (it's sbo not sov), (or so I thought at the time, see deemok's edits below) so I tried debugging with the following configuration:

<ADPlus>
   <!-- Add log entry, log faulting thread stack and dump full on first chance StackOverflow -->
<Exceptions>
     <Config>
        <!-- This is for the StackOverflow exception -->
       <Code> sbo </Code>
       <Actions1> Log;Stack;FullDump </Actions1>
       <!-- Depending on what you intend - either stop the debugger (Q or QQ) or continue unhandled (GN) -->
       <ReturnAction1> GN </ReturnAction1>
     </Config>
  </Exceptions>
</ADPlus>

And using the following command:

adplus -crash -o D:\Crash -NoDumpOnFirst -c D:\Crash\stackoverflow.cfg -iis

I verified that the outputted log files indictated the right configuration. The trick is that the command line params of adplus get executed in order, so if you start with a config that traps first-chance exceptions and then apply -NoDumpOnFirst, the configuration settings will be overwritten. If you apply the config with -c last, then its settings will win out.

In the end, however, the stack overflow proved uncatchable. The stack overflow happened, no memory dump could be received, and then a dump occurred on the second-chance process ending event, and again everything had been garbage collected and I couldn't get any useful information.

I attempted to short-circuit the process ending exception, in case that was engaging and overriding the stack overflow, but then the exception occurred and I just got no memory dump.

Luckily, I stumbled upon the answer by examining code. It was a case of circular method calling, of course.

Actual Resolution

The problem had been solved long ago but I quickly made an ASP.NET page that would cause a stack overflow. (It's not hard to do after all) and tried Axl's response below.

The XML was slightly off - Axl just forgot to close the </ADPlus> tag (or probaby lost it in a copy-paste), but that was easy enough to fix and adplus was kind enough to tell me exactly what was wrong.

I set that script off against my test stack overflow thrower, loaded up the result in windbg, and when I called !clrstack I got a very clear (and long) listing of the methods that were calling each other circularly. This would have found the problem in an instant! I'll be keeping this page bookmarked for the next time a stack overflow comes knocking at my door.

Answer 1

Just in case this might help someone else, below is the ADPlus config file I came up with. Looking at it now, I'm not sure that !runaway has any effect. Attached when an ASP.NET app that throws a StackOverflowException is running, this will generate "1st chance StackOverflow full" and "1st chance Process Shut Down full" .dmp files in the specified OutputDir. Open the first file with Windbg, and run ".loadby sos mscorwks" followed by "!clrstack" to see what might be causing the stack overflow.

<ADPlus>
<Settings>
    <RunMode>CRASH</RunMode>
    <OutputDir>C:\Dumps</OutputDir>
    <ProcessName>w3wp.exe</ProcessName> 
</Settings>
<Exceptions>
    <Option>FullDumpOnFirstChance</Option>
    <Option>MiniDumpOnSecondChance</Option>
    <Option>NoDumpOnFirstChance</Option>
    <Option>NoDumpOnSecondChance</Option>
    <Config>
        <Code>AllExceptions</Code>
        <Actions1>Void</Actions1>
        <Actions2>Void</Actions2>
        <ReturnAction1>GN</ReturnAction1>
        <ReturnAction2>GN</ReturnAction2>
    </Config>       
    <Config>
        <!--
        av = AccessViolation
        ch = InvalidHandle
        ii = IllegalInstruction
        dz =  IntegerDivide
        c000008e = FloatingDivide
        iov = IntegerOverflow
        lsq = InvalidLockSequence
        sov = StackOverflowException
        eh = CPlusPlusEH
        * = UnknownException
        clr = NET_CLR
        bpe = CONTRL_C_OR_Debug_Break
        ld = DLL_Load
        ud = DLL_UnLoad
        epr = Process_Shut_Down
        sbo = Stack_buffer_overflow
        -->
        <Code>sov;sbo</Code>
        <Actions1>Log;Time;Stack;FullDump;EventLog</Actions1>
        <CustomActions1>!runaway</CustomActions1>
        <Actions2>Log;Time;Stack;FullDump;EventLog</Actions2>
        <CustomActions2>!runaway</CustomActions2>
        <!--
        G = go
        GN = go unhandled exception
        GH = go handled exception
        Q = quit
        QD = quit and detach
        -->
        <ReturnAction1>GN</ReturnAction1>
        <ReturnAction2>GN</ReturnAction2>
    </Config>
    <Config>
        <Code>clr</Code>
        <Actions1>Void</Actions1>
        <Actions2>Log;Time;Stack;FullDump;EventLog</Actions2>
        <ReturnAction1>GN</ReturnAction1>
        <ReturnAction2>GN</ReturnAction2>
    </Config>
    <Config>
        <Code>epr</Code>
        <Actions1>Log;Time;Stack;FullDump;EventLog</Actions1>
        <Actions2>Void</Actions2>
        <ReturnAction1>GN</ReturnAction1>
        <ReturnAction2>GN</ReturnAction2>
    </Config>
</Exceptions>
</ADPlus>

Answer 2

<ADPlus>
   <!-- Add log entry, log faulting thread stack and dump full on first chance StackOverflow -->
<Exceptions>
     <Config>
        <!-- This is for the stack buffer overflow exception -->
        <!-- Use sov for stack overflow exception -->
       <Code> sbo </Code>
       <Actions1> Log;Stack;FullDump </Actions1>
       <!-- Depending on what you intend - either stop the debugger (Q or QQ) or continue unhandled (GN) -->
       <ReturnAction1> GN </ReturnAction1>
     < Config>
  </Exceptions>
</ADPlus>

Save that in stackoverflow.cfg
Then you can go:

adplus -c stackoverflow.cfg

Edit: both sov and sbo are stack overflow exceptions. I guess one needs to experiment with both since it is not quite clear to me what the difference between the two is. (might sbo denote an invalid alloca() call?)