Why does a .NET program survive a corrupt stack? (when using the wrong calling convention)

Question

In VS2010, the managed debugging assistant will give you a pInvokeStackImbalance exception (pInvokeStackImbalance MDA) if you call a function using the wrong calling convention, commonly because you didn't specify CallingConvention = Cdecl when calling a C library. E.g. you wrote

[DllImport("some_c_lib.dll")]
static extern void my_c_function(int arg1, int arg2);

instead of

[DllImport("some_c_lib.dll", CallingConvention=CallingConvention.Cdecl)]
static extern void my_c_function(int arg1, int arg2);

and thus got the StdCall calling convention instead of Cdelc.

If you answer this, you already know the difference, but for other visitors to this thread: StdCall means that the callee cleans the arguments from the stack, whereas Cdecl means that the caller cleans up the stack.

So, if you get the calling convention wrong in your C code, your stack doesn't get cleaned up and your program crashes.

However, .NET programs don't seem to crash even though they use StdCall for Cdecl functions. The stack imbalance check wasn't enabled by default on VS2008, so some VS2008 projects use the wrong calling convention unbeknownst to their authors. I just tried GnuMpDotNet and the sample runs just fine even though the Cdelc declaration is missing. The same holds true for X-MPIR.

They both throw the pInvokeStackImbalance MDA exception in debug mode, but don't crash in release mode. Why is this? Does the .NET VM wrap all calls to native code and restore the stack itself afterwards? If so, why bother with the CallingConvention property at all?

Answer 1

It is because of the way the stack pointer is restored when the method exits. The standard prologue of a method, shown for the x86 jitter;

00000000  push        ebp                 ; save old base pointer
00000001  mov         ebp,esp             ; setup base pointer to point to activation frame
00000003  sub         esp,34h             ; reserve space for local variables

And the way it ends:

0000014a  mov         esp,ebp             ; restore stack pointer
0000014c  pop         ebp                 ; restore base pointer
0000014d  ret 

Getting the esp value unbalanced is not a problem here, it gets restored from the ebp register value. However, the jitter optimizer not infrequently optimizes this away when it can store local variables in cpu registers. You'll crash and burn when the RET instruction retrieves the wrong return address from the stack. Hopefully anyway, really nasty when it happens to land on a chunk of machine code by accident.

This is liable to happen when you run the release build without a debugger, tough to troubleshoot if you didn't have the MDA to help you.