How can a cdecl calling convention corrupt the ESP?

Question

My application is crashing because a library function that I call changes the ESP, though it is declared as cdecl.

The library (libclang.dll) is compiled using MinGW and I am using it from a VC++ project. The functions are exported as C-functions and Dependency Walker tells me they have the correct cdecl calling convention. The functions are imported in my project using dllimport by including Clang's "index.h" file. It seems, not all functions are corrupting the ESP, thus some functions perform successful, others lead to crashes.

Here is the assembly of a working function:

// call to clang_getNumDiagnostics(TU); - works!
5AF3EFAB  mov         esi,esp  
5AF3EFAD  mov         eax,dword ptr [ebp-30h]  
5AF3EFB0  push        eax  
5AF3EFB1  call        dword ptr [__imp__clang_getNumDiagnostics (5AF977E0h)]  
5AF3EFB7  add         esp,4  
5AF3EFBA  cmp         esi,esp  
5AF3EFBC  call        @ILT+7135(__RTC_CheckEsp) (5AF16BE4h)

The following function call will change esp (adding 4) and thus lead to a crash due to the runtime check in __RTC_CheckEsp.

// call to clang_getTranslationUnitCursor(TU); - fails!
5AF3EFC1  mov         esi,esp  
5AF3EFC3  mov         eax,dword ptr [ebp-30h]  
5AF3EFC6  push        eax  
5AF3EFC7  lea         ecx,[ebp-234h]  
5AF3EFCD  push        ecx  
5AF3EFCE  call        dword ptr [__imp__clang_getTranslationUnitCursor (5AF9780Ch)]  
5AF3EFD4  add         esp,8  
5AF3EFD7  cmp         esi,esp  
5AF3EFD9  call        @ILT+7135(__RTC_CheckEsp) (5AF16BE4h)  

I already posted a question for this problem, but thought I asked specifically about the calling convention cdecl to retrieve more specific information about the possibility of a currupted esp as I think this may be the source of the problem ... Thus excuse this "double post".

The source could also lie in the wrong function being called (maybe due to a problem in the def file that i created with dlltool and later created the import lib from - the ordinals differed from what Dependency Walker showed though - I tried it with corrected ordinals, but no change). I feel this is unlikely the problem source though, because the other function calls work fine and return correct values ...

Thanks!

[Update]

As requested the assembly for __imp__clang_getTranslationUnitCursor

6660A4A0  push        ebp  
6660A4A1  mov         ebp,esp  
6660A4A3  push        edi  
6660A4A4  push        ebx  
6660A4A5  mov         eax,dword ptr [ebp+8]  
6660A4A8  mov         ebx,eax  
6660A4AA  mov         al,0  
6660A4AC  mov         edx,14h  
6660A4B1  mov         edi,ebx  
6660A4B3  mov         ecx,edx  
6660A4B5  rep stos    byte ptr es:[edi]  
6660A4B7  mov         eax,dword ptr [ebp+8]  
6660A4BA  mov         dword ptr [eax],12Ch  
6660A4C0  mov         eax,dword ptr [ebp+8]  
6660A4C3  mov         edx,dword ptr [ebp+0Ch]  
6660A4C6  mov         dword ptr [eax+10h],edx  
6660A4C9  mov         eax,dword ptr [ebp+8]  
6660A4CC  pop         ebx  
6660A4CD  pop         edi  
6660A4CE  pop         ebp  
6660A4CF  ret         4  

[Update 2] As both VC++ and GCC use cdecl as default and there is no way to force another default calling convention in GCC without explicitly stating it in the function declaration (which is not done for the problematic functions), I am actually sure that cdecl is used everywhere.

I found this link though, which states some differences that may explain why some functions work and others don't:

Visual C++ / Win32

• Objects larger than 8 bytes are returned in memory.

• When a return is made in memory the caller passes a pointer to the memory location as the first parameter (hidden). The callee populates the memory, and returns the pointer. The caller pops the hidden pointer together with the rest of the arguments.

MinGW g++ / Win32

• Objects larger than 8 bytes are returned in memory.

• When a return is made in memory the caller passes a pointer to the memory location as the first parameter (hidden). The callee populates the memory, and returns the pointer. The callee pops the hidden pointer from the stack when returning.

may that be the problem? is there any way I can solve this problem? or do i have to alter Clang's Index.h and switch to stdCall?

[Update 3]

Here is the according GCC-Bug. Seems that in 4.6 (64bit) and 4.7 (32bit) you can use the new ms_abi function attribute to fix the issue described in [Update 2].

Answer 1

GCC and Visual C++ do not implement the same cdecl calling convention. Wikipedia explains:

There are some variations in the interpretation of cdecl, particularly in how to return values. As a result, x86 programs compiled for different operating system platforms and/or by different compilers can be incompatible, even if they both use the "cdecl" convention and do not call out to the underlying environment. [...] To pass "in memory", the caller allocates memory and passes a pointer to it as a hidden first parameter; the callee populates the memory and returns the pointer, popping the hidden pointer when returning.

The last sentence is the important one: GCC's version of cdecl makes the callee clean the hidden pointer, whereas Visual C++'s version of cdecl leaves it for the caller to clean.