What happens if i overwrite the return address on the stack?

Question

I know this is dangerous behaviour, but I would like to figure out what happened.

Here is the code:

#include<stdio.h>
#include<stdlib.h>

static int count = 0;

void hello(void){
    count ++;  
    fprintf(stderr,"hello! %d\n",count);
}

void foo(void){
    void *buf[10];
    static int i;

    for(i = 0 ; i < 100 ; i++){ // put enough data to overwrite the return address on the stack
        buf[i] = hello;
    }
}

int main(void){
    int buf[1000];
    foo();
    return 0;
}

And here is the result:

……
hello! 83
hello! 84
hello! 85
hello! 86
hello! 87
hello! 88
hello! 89
Segmentation fault (core dumped)

Why has the hello function been called 89 times? When the function foo returns, the pc register should get the address of hello function, shouldn't it? So hello is called and the code inside is executed, then what? Shouldn't the program return to main function? Where does the "89" came from? I seem to be missing something, could someone please point it out to me .

Answer 1

You wrote the address of the function hello to the stack of foo 90 times beyond the amount of space you reserved with buf. When foo tried to return, it loaded (one of) the first extra address(es) of hello into the program counter, popped it, and "returned" to it. The reason I say "one of" is because there may be some space on the stack between the end of the array you reserved and the beginning of the return address.

hello assumed that it was called by foo. However, it was not called, in other words, the address of the remainder of foo was not pushed onto the stack for hello to return to. So, when hello returned, the thing it popped was the next thing on the stack, which was again the address of hello.

This again looked like a call from hello's point of view, but it was not actually a call, since it was a return, so hello kept returning to itself and popping one address of hello from the stack each time, until it ran out of such addresses.

After those ran out, the next bytes on the stack, which you had not overwritten, when treated as a pointer, were to elsewhere in memory, probably to something that was not allocated to your process memory map. hello tried to return to that, and that's when the program segfaulted.

Answer 2

Well this is undefined behavior. When you step into unallocated memory anything can happen depending on what "junk" there is at the moment.

Segmentation fault basically means that you stepped out of the protected memory of your program (in your case it took "89 times"). Your operating system defends you against this and tells you that you tried to write outside of the space allocated for the program.

(This was not the case in old operating systems and such a bug could cause your operating system to crash if your program overflowed into system space.)