Why does using the wrong calling convention sometimes work?

Question

I used "StartServiceCtrlDispatcher" function to register a callback function (called ServiceMain) in windows, but the callback function I declared got compiled with the wrong calling convention.

The thing is that on some computers, when the application returned from the callback function, the application crashed, but on other computers the application did not crash.

Now, once I found the bug everything worked, but I just don't understand why on some computers it worked correctly without crashing ?

Thanks! :-)

Answer 1

This is all very Windows-specific, we're not talking standard C++ here.

Checking out the documentation of StartServiceDispatcher it has only one argument, and is declared as WINAPI which in turn means __stcall calling convention.

For freestanding functions, __stdcall is one of two main calling conventions. The other one is __cdecl. The machine code level difference is simply who restores the stack pointer: with __stdcall it is the function itself, while with __cdecl it is the calling code.

When the function actually is __stdcall but is invoked as if it was __cdecl, the situation is that there are two attempts to restore the stack pointer: one at the exit from the function, and one in the calling code. The one in the function will succeed. Depending on how the attempt in the calling code is done, it can mess things up thoroughly (e.g. if just adding the required offset, treating the stack pointer as relative), or it may have no harmful effect. But it's very likely to create a mess, since the assumption about the stack pointer value on return from the function, is incorrect.

When the function actually is __cdecl it will not itself restore the stack pointer, since that is the calling code's responsibility. And if the calling code is treating it as __stdcall then the calling code won't restore it either, since from the calling code's view the function is doing that. The result, if you don't get an early crash (because of broken assumptions), should then be that repeated calls, say in a loop, will eat stack space.

It's all very much Undefined Behavior.

And one property of Undefined Behavior is that it can do anything, including apparently working…

Cheers & hth.,

Answer 2

Calling conventions differ in the details, like which registers are preserved. If you happened to not store anything you still needed in those registers, then it didn't matter that they were erased when they didn't have to be. Similarly, if your calling convention differs about how it deals with return values, if you don't return anything, then it doesn't matter.

Fortunately, x64 only has one calling convention and this whole mess will be in the past.