C++ Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call

Question

I am trying to program motorbee using c++

when I run the code I get the following error:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

This is my code.

#include "stdafx.h"
#include <iostream>
#include "windows.h"
#include "mt.h"
using namespace std;

HINSTANCE BeeHandle= LoadLibrary("mtb.dll"); 
Type_InitMotoBee InitMotoBee;
Type_SetMotors SetMotors;
Type_Digital_IO Digital_IO;
void main ()
{   
    InitMotoBee = (Type_InitMotoBee)GetProcAddress( BeeHandle,"InitMotoBee");
    SetMotors =(Type_SetMotors)GetProcAddress(BeeHandle,"SetMotors");
    Digital_IO =(Type_Digital_IO)GetProcAddress(BeeHandle,"Digital_IO ");
    InitMotoBee();

    SetMotors(0, 50, 0, 0, 0, 0, 0, 0, 0);

}

Answer 1

Your typedef function pointer needs to match the calling convention of the library you are using. For example, if InitMotoBee uses cdecl your typedef will look like:

typedef bool (__cdecl *Type_InitMotoBee)(void)

The SetMotors function takes parameters so the calling convention will need to be set correctly for that as well (that is likely where the application is failing).

Answer 2

The error message is telling you that the ESP register (Stack Pointer) has not been properly "maintained". It doesn't have the value it should have.

When you make a function call in an unmanaged language like C or C++ the arguments to the function are pushed on to the stack - increasing the stack pointer. When the function call returns, the arguments are popped back off - decreasing the stack pointer.

The stack pointer must always be restored to the same value it had before the function call.

Calling Conventions

A calling convention specifies precisely how the stack should be maintained, and whether the caller or callee is responsible for popping the arguments off the stack.

For example, in the stdcall calling convention the callee is responsible for restoring the stack pointer before the function returns. In the cdecl calling convention, the caller is responsible.

It should be obvious that mixing calling conventions is bad! If the caller is using stdcall, it's expecting the callee to maintain the stack. If the callee is using cdecl, it's expecting the caller to maintain the stack. End result: no one is maintaining the stack! Or the opposite example: everyone is maintaining the stack, meaning it gets restored twice and ends up wrong.

For reference, look at this StackOverflow question.

Raymond Chen has a good blog post on the subject.

Which Calling Convention Should You Use?

That's beyond the scope of this answer, but if you're doing C# to C interop, it is important to know what calling conventions are in place.

In Visual Studio, the default calling convention for a C/C++ project is cdecl.

In .Net the default calling convention for interop calls using DllImport is stdcall. This applies to delegates too. (Most native Windows functions use stdcall.)

Consider the following (incorrect) interop call.

[DllImport("MyDll", EntryPoint = "MyDll_Init"]
public static extern void Init();

It is using the stdcall calling convention, because that is .Net's default. If you haven't changed the Visual Studio project settings for your MyDLL project, you'll soon find this doesn't work. The default for a C/C++ DLL project is cdecl.

The correct interop call would be:

[DllImport("MyDll", EntryPoint = "MyDll_Init", CallingConvention = CallingConvention.Cdecl)]
public static extern void Init();

Note the explicit CallingConvention attribute. The C# interop wrapper will know to generate a cdecl call.

What else can go wrong?

If you're sure your calling conventions are correct, you might still encounter run-time check failure #0.

Marshalling Structs

Recall that the function arguments are pushed on to the stack at the start of a function call, then popped off again at the end. In order to ensure that the stack is correctly maintained, the sizes of the arguments must be consistent between the push and pop.

In native code, the compiler will deal with this for you. You never need to think about. When it comes to interop between C and C#, you might get bitten.

If you have a stdcall delegate in C#, something like this:

public delegate void SampleTimeChangedCallback(SampleTime sampleTime);

which corresponds to a C function pointer, something like this:

typedef void(__stdcall *SampleTimeChangedCallback)(SampleTime sampleTime);

everything should be fine. You're using the same calling convention on both sides (C# interop uses stdcall by default, and we're explicitly setting __stdcall in the native code).

But look at those parameters: the SampleTime struct. They both have the same name, but one is a native struct, the other is a C# struct.

The native struct looks something like this:

struct SampleTime
{
    __int64 displayTime;
    __int64 playbackTime;
}

The C# struct looks like this:

[StructLayout(LayoutKind.Explicit, Size = 32)]
public struct SampleTime
{
    [FieldOffset(0)]
    private long displayTime;

    [FieldOffset(8)]
    private long playbackTime;
}

Look at the Size attribute on the C# struct - it's wrong! Two 8-byte longs means a 16-byte size. Perhaps someone has removed some fields and failed to update the Size attribute.

Now, when the native code calls the SampleTimeChangedCallback function, using stdcall, we run into a problem.

Recall that in stdcall, the callee - i.e. the function being called - is responsible for restoring the stack.

So: the caller pushes parameters on to the stack. In this example, that's happening in native code. The size of the parameters is known by the compiler, so value by which the stack pointer is incremented is guaranteed to be correct.

The function then executes - remember that in reality this is a c# delegate.

Since we're using stdcall, the callee - the c# delegate - is responsible for restoring the stack. But in C# land we have lied to the compiler and told it that the size of the SampleTime structure is 32 bytes when it's really only 16.

We've violated the One Definition Rule.

The C# compiler has no option but to believe what we tell it, so it will "restore" the stack pointer by 32bytes.

When we return back to the callsite (in native land) the stack pointer has NOT been properly restored, and all bets are off.

If you are lucky, you'll encounter a run-time check #0. If you're unlucky the program might not crash straight away. The one thing you can be sure of: your program is no longer executing the code you thought it was.

Answer 3

I ended up changing the compiler option from /RTC1 (which is effectively both /RTCs and /RTCu) to /RTCu. http://support.microsoft.com/kb/822039