Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Understanding the C function call prolog with __cdecl on windows

Compiling this simple function with MSVC2008, in Debug mode:

int __cdecl sum(int a, int b)
{
    return a + b;
}

I get the following disassembly listing:

int __cdecl sum(int a, int b)
{
004113B0  push        ebp  
004113B1  mov         ebp,esp 
004113B3  sub         esp,0C0h 
004113B9  push        ebx  
004113BA  push        esi  
004113BB  push        edi  
004113BC  lea         edi,[ebp-0C0h] 
004113C2  mov         ecx,30h 
004113C7  mov         eax,0CCCCCCCCh 
004113CC  rep stos    dword ptr es:[edi] 
    return a + b;
004113CE  mov         eax,dword ptr [a] 
004113D1  add         eax,dword ptr [b] 
}
004113D4  pop         edi  
004113D5  pop         esi  
004113D6  pop         ebx  
004113D7  mov         esp,ebp 
004113D9  pop         ebp  
004113DA  ret

There are some parts of the prolog I don't understand:

004113BC  lea         edi,[ebp-0C0h] 
004113C2  mov         ecx,30h 
004113C7  mov         eax,0CCCCCCCCh 
004113CC  rep stos    dword ptr es:[edi] 

Why is this required?


EDIT:

After removing the /RTC compiler option, as was suggested, most of this code indeed went away. What remained is:

int __cdecl sum(int a, int b)
{
00411270  push        ebp  
00411271  mov         ebp,esp 
00411273  sub         esp,40h 
00411276  push        ebx  
00411277  push        esi  
00411278  push        edi  
    return a + b;
00411279  mov         eax,dword ptr [a] 
0041127C  add         eax,dword ptr [b] 
}

Now, why is the: sub esp, 40h needed? It's as if place is being allocated for local variables, though there aren't any. Why is the compiler doing this? Is there another flag involved?

like image 730
Eli Bendersky Avatar asked Jan 16 '10 10:01

Eli Bendersky


2 Answers

This code is emitted due to the /RTC compile option. It initializes all local variables in your function to a bit pattern that is highly likely to generate an access violation or to cause unusual output values. That helps you find out when you forgot to initialize a variable.


The extra space in the stack frame you see allocated is there to support the Edit + Continue feature. This space will be used when you edit the function while debugging and add more local variables. Change the /ZI option to /Zi to disable it.

like image 187
Hans Passant Avatar answered Oct 21 '22 06:10

Hans Passant


and in any case of buffer overflow (if you would overwrite local variables) you will end up in a field of "int 3" opcodes:

int 3 ; 0xCC
int 3 ; 0xCC
int 3 ; 0xCC
int 3 ; 0xCC
int 3 ; 0xCC
int 3 ; 0xCC
...

that can be catched by the debugger, so you can fix your code

like image 35
Bartosz Wójcik Avatar answered Oct 21 '22 06:10

Bartosz Wójcik