Why is there no "sub rsp" instruction in this function prologue and why are function parameters stored at negative rbp offsets?

Question

That's what I understood by reading some memory segmentation documents: when a function is called, there are a few instructions (called function prologue) that save the frame pointer on the stack, copy the value of the stack pointer into the base pointer and save some memory for local variables.

Here's a trivial code I am trying to debug using GDB:

void test_function(int a, int b, int c, int d) {
    int flag;
    char buffer[10];

    flag = 31337;
    buffer[0] = 'A';
}

int main() {
    test_function(1, 2, 3, 4);
}

The purpose of debugging this code was to understand what happens in the stack when a function is called: so I had to examine the memory at various step of the execution of the program (before calling the function and during its execution). Although I managed to see things like the return address and the saved frame pointer by examining the base pointer, I really can't understand what I'm going to write after the disassembled code.

Disassembling:

(gdb) disassemble main
Dump of assembler code for function main:
   0x0000000000400509 <+0>: push   rbp
   0x000000000040050a <+1>: mov    rbp,rsp
   0x000000000040050d <+4>: mov    ecx,0x4
   0x0000000000400512 <+9>: mov    edx,0x3
   0x0000000000400517 <+14>:    mov    esi,0x2
   0x000000000040051c <+19>:    mov    edi,0x1
   0x0000000000400521 <+24>:    call   0x4004ec <test_function>
   0x0000000000400526 <+29>:    pop    rbp
   0x0000000000400527 <+30>:    ret    
End of assembler dump.
(gdb) disassemble test_function 
Dump of assembler code for function test_function:
   0x00000000004004ec <+0>: push   rbp
   0x00000000004004ed <+1>: mov    rbp,rsp
   0x00000000004004f0 <+4>: mov    DWORD PTR [rbp-0x14],edi
   0x00000000004004f3 <+7>: mov    DWORD PTR [rbp-0x18],esi
   0x00000000004004f6 <+10>:    mov    DWORD PTR [rbp-0x1c],edx
   0x00000000004004f9 <+13>:    mov    DWORD PTR [rbp-0x20],ecx
   0x00000000004004fc <+16>:    mov    DWORD PTR [rbp-0x4],0x7a69
   0x0000000000400503 <+23>:    mov    BYTE PTR [rbp-0x10],0x41
   0x0000000000400507 <+27>:    pop    rbp
   0x0000000000400508 <+28>:    ret    
End of assembler dump.

I understand that "saving the frame pointer on the stack" is done by " push rbp", "copying the value of the stack pointer into the base pointer" is done by "mov rbp, rsp" but what is getting me confused is the lack of a "sub rsp $n_bytes" for "saving some memory for local variables". I've seen that in a lot of exhibits (even in some topics here on stackoverflow).

I also read that arguments should have a positive offset from the base pointer (after it's filled with the stack pointer value), since if they are located in the caller function and the stack grows toward lower addresses it makes perfect sense that when the base pointer is updated with the stack pointer value the compiler goes back in the stack by adding some positive numbers. But my code seems to store them in a negative offset, just like local variables.. I also can't understand why they are put in those registers (in the main).. shouldn't they be saved directly in the rsp "offsetted"?

Maybe these differences are due to the fact that I'm using a 64 bit system, but my researches didn't lead me to anything that would explain what I am facing.

Answer 1

The System V ABI for x86-64 specifies a red zone of 128 bytes below %rsp. These 128 bytes belong to the function as long as it doesn't call any other function (it is a leaf function).

Signal handlers (and functions called by a debugger) need to respect the red zone, since they are effectively involuntary function calls.
All of the local variables of your test_function, which is a leaf function, fit into the red zone, thus no adjustment of %rsp is needed. (Also, the function has no visible side-effects and would be optimized out on any reasonable optimization setting).

You can compile with -mno-red-zone to stop the compiler from using space below the stack pointer. Kernel code has to do this because hardware interrupts don't implement a red-zone.

Answer 2

But my code seems to store them in a negative offset, just like local variables

The first x86_64 arguments are passed on registers, not on the stack. So when rbp is set to rsp, they are not on the stack, and cannot be on a positive offset.

They are being pushed only to:

• save register state for a second function call.

  In this case, this is not required since it is a leaf function.

• make register allocation easier.

  But an optimized allocator could do a better job without memory spill here.

The situation would be different if you had:

• x86_64 function with lots of arguments. Those that don't fit on registers go on the stack.
• IA-32, where every argument goes on the stack.

the lack of a "sub rsp $n_bytes" for "saving some memory for local variables".

The missing sub rsp on red zone of leaf function part of the question had already been asked at: Why does the x86-64 GCC function prologue allocate less stack than the local variables?