gcc stack optimization

Question

Hi I have a question on possible stack optimization by gcc (or g++)..

Sample code under FreeBSD (does UNIX variance matter here?):

void main() {
   char bing[100];
   ..
   string buffer = ....;
   ..
}

What I found in gdb for a coredump of this program is that the address of bing is actually lower than that buffer (namely, &bing[0] < &buffer).

I think this is totally the contrary of was told in textbook. Could there be some compiler optimization that re-organize the stack layout in such a way?

This seems to be only possible explanation but I'm not sure..

In case you're interested, the coredump is due to the buffer overflow by bing to buffer (but that also confirms &bing[0] < &buffer).

Thanks!

Answer 1

Compilers are free to organise stack frames (assuming they even use stacks) any way they wish.

They may do it for alignment reasons, or for performance reasons, or for no reason at all. You would be unwise to assume any specific order.

If you hadn't invoked undefined behavior by overflowing the buffer, you probably never would have known, and that's the way it should be.

A compiler can not only re-organise your variables, it can optimise them out of existence if it can establish they're not used. With the code:

#include <stdio.h>
int main (void) {
   char bing[71];
   int x = 7;
   bing[0] = 11;
   return 0;
}

Compare the normal assembler output:

main:
    pushl   %ebp
    movl    %esp, %ebp
    andl    $-16, %esp
    subl    $80, %esp
    movl    %gs:20, %eax
    movl    %eax, 76(%esp)
    xorl    %eax, %eax
    movl    $7, (%esp)
    movb    $11, 5(%esp)
    movl    $0, %eax
    movl    76(%esp), %edx
    xorl    %gs:20, %edx
    je      .L3
    call    __stack_chk_fail
.L3:
    leave
    ret

with the insanely optimised:

main:
    pushl   %ebp
    xorl    %eax, %eax
    movl    %esp, %ebp
    popl    %ebp
    ret

Notice anything missing from the latter? Yes, there are no stack manipulations to create space for either bing or x. They don't exist. In fact, the entire code sequence boils down to:

• set return code to 0.
• return.

Answer 2

A compiler is free to layout local variables on the stack (or keep them in register or do something else with them) however it sees fit: the C and C++ language standards don't say anything about these implementation details, and neither does POSIX or UNIX. I doubt that your textbook told you otherwise, and if it did, I would look for a new textbook.