Why isn't PInvoke crashing in case of violated calling convention (in .NET 3.5)?

Question

My solution has an unmanaged C++ DLL, which exports a function, and a managed application that PInvokes this function.

I've just converted the solution from .NET 3.5 to .NET 4.0 and got this PInvokeStackImbalance "A call to PInvoke function [...] has unbalanced the stack" exception. As it turned out, I was calling __cdecl'ed function, as it was __stdcall:

C++ part (callee):

__declspec(dllexport) double TestFunction(int param1, int param2); // by default is __cdecl

C# part (caller):

[DllImport("TestLib.dll")] // by default is CallingConvention.StdCall
private static extern double TestFunction(int param1, int param2);

So, I've fixed the bug, but now I'm interested in how did this work in .NET 3.5? Why the (many times repeated) situation when nobody (neither callee, nor caller) cleans the stack, did not caused stack overflow or some other misbehavior, but just worked OK? Is there some sort of a check in PInvoke, like mentioned by Raymond Chen in his article? It's also interesting, why the opposite type of breaking convention (having __stdcall callee be PInvoked like being __cdecl) is not working at all, causing just EntryPointNotFoundException.

Answer 1

PInvokeStackImbalance is not an exception. It is an MDA warning, implemented by a Managed Debugging Assistant. Having that MDA active is optional, you can configure that from the Debug + Exceptions dialog. It will never be active when you run without a debugger.

Getting the stack imbalanced can cause pretty nasty problems, ranging from strange data corruption to getting SOE or AVE. Very hard to diagnose too. But it can also cause no trouble at all, the stack pointer gets restored when the method returns.

Code compiled to 64-bit tends to be resilient, a lot more of the function arguments get passed through registers instead of the stack. It will fail when forced to run on x86, the new default for VS2010.