Free-running application under test exits shortly after I attach with VS2010 SP1 in x86

Question

On Windows 7 x64, when I attach in x86 mode to a fairly complex free-running app, it runs for a while, then reproducibly exits.

MyApp.exe Managed (v4.0.30319)' has exited with code -1073740791 (0xc0000409).

followed immediately by

MyApp.vshost.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).

Sometimes if it runs OK, it would hit my breakpoint, I'll inspect the state, but when I hit F5 to keep going, the app exits in the same fashion.

Quick search for the error code tells me that it's a Stack Buffer Overrun. I hear that it might be caused by incorrect unmanaged interop code.

I can run from debugger OK (F5), but free-running and attaching always has this problem.

Any thoughts on how I could narrow it down?

EDIT: Here's a callstack i am seeing on a different machine (Windows Server 2008 R2 x64) here, might be related:

clr.dll!__crt_debugger_hook()
clr.dll!___report_gsfailure() + 0xeb bytes clr.dll!_DoJITFailFast@0() + 0x8 bytes clr.dll!CrawlFrame::SetCurGSCookie() + 0x2e9c4f bytes
clr.dll!StackFrameIterator::Init() + 0x60 bytes
clr.dll!Thread::StackWalkFramesEx() + 0x8a bytes
clr.dll!Thread::StackWalkFrames() + 0x87 bytes clr.dll!CNameSpace::GcScanRoots() + 0xd7 bytes clr.dll!WKS::gc_heap::mark_phase() + 0xae bytes
clr.dll!WKS::gc_heap::gc1() + 0x7b bytes
clr.dll!WKS::gc_heap::garbage_collect() + 0x1c1 bytes
clr.dll!WKS::GCHeap::GarbageCollectGeneration() + 0xba bytes
clr.dll!WKS::gc_heap::try_allocate_more_space() + 0x1cd0 bytes clr.dll!WKS::gc_heap::allocate_more_space() + 0x13 bytes
clr.dll!WKS::GCHeap::Alloc() + 0x507 bytes clr.dll!Alloc() + 0x5a bytes
clr.dll!SlowAllocateString() + 0x41 bytes
clr.dll!UnframedAllocateString() + 0x11 bytes
clr.dll!StringObject::NewString() + 0x26 bytes clr.dll!Int64ToDecStr() + 0x12e bytes
clr.dll!COMNumber::FormatInt64() + 0x17e bytes mscorlib.ni.dll!6c60b8e1()
[Frames below may be incorrect and/or missing, no symbols loaded for mscorlib.ni.dll]

EDIT2 Things seem fine on x64 build of the app, issue only appears in x86.

Answer 1

From the Windows SDK ntstatus.h header file:

//
// MessageId: STATUS_STACK_BUFFER_OVERRUN
//
// MessageText:
//
// The system detected an overrun of a stack-based buffer in this application. This overrun 
// could potentially allow a malicious user to gain control of this application.
//
#define STATUS_STACK_BUFFER_OVERRUN      ((NTSTATUS)0xC0000409L)    // winnt

A buffer overrun on a stack allocated buffer is an infamous virus injection vector. Microsoft got very serious about eliminating that potential thread in their code. The C and C++ languages were first. Managed code straggled behind, this is not something that is supposed to happen in a managed execution environment.

Nevertheless, the version 4 CLR was built with the protection in place, unlike earlier CLR versions. And it does its job, although it is exceedingly rare for it to happen. I've seen a question about it only once before.

Solving this problem is going to be difficult, especially when you have no obvious lead to what unmanaged code in your application might be tripping this protection. Best thing to do is to make a minimal repro and contact Microsoft Support to show them what is going wrong. Finding out what trips it while working on getting the repro is a likely outcome.

Answer 2

is the interop signature correct? try using http://clrinterop.codeplex.com/releases/view/14120 to generate it and try again.

Answer 3

It looks like I should be able to narrow it down by injecting GC.Collect() calls in my code: garbage collection checks GSCookie among other things.

Broken link1: http://7388.info/index.php/article/studio/2010-10-17/354.html

Broken link2: http://www.pubsub.com/Investigating-a-GSCookie-Corruption_Windows-NET-Troubleshooting-PInvoke-5wbEHu80dzF,rZ5U5DaVJaE