What Happens When Stack and Heap Collide

Question

I am curious to know what happens when the stack and the heap collide. If anybody has encountered this, please could they explain the scenario.

Answer 1

In a modern languages running on a modern OS, you'll get either a stack overflow (hurray!) or malloc() or sbrk() or mmap() will fail when you try to grow the heap. But not all software is modern, so let's look at the failure modes:

• If the stack grows into the heap, the typically C compiler will silently start to overwrite the heap's data structures. On a modern OS, there will be one or more virtual memory guard pages which prevent the stack from growing indefinitely. As long as the amount of memory in the guard pages is at least as large as the size of the growing procedure's activation record, the OS will guarantee you a segfault. If you're DOS running on a machine with no MMU, you're probably hosed.

• If the heap grows into the stack, the operating system should always be aware of the situation and some sort of system call will fail. The implementation of malloc() almost certainly notices the failure and returns NULL. What happens after that is up to you.

I'm always amazed at the willingness of compiler writers to hope that the OS puts guard pages in place to prevent stack overflow. Of course, this trick works well until you start having thousands of threads, each with its own stack...

Answer 2

This would be platform dependent. On many platforms it actually can't happen at all (the heap and stack are allocated in different pages and ne'r the twain shall meet.

Keep in mind that the idea of the heap growing upward and the stack growing downward is only conceptual. On very small systems (like the old 8-bit micros that ran CP/M) and on some PICs and other flat memory model systems (those without an MMU nor any other virtual or protected memory support) then the heap and stack might be actually implemented this way. In that case the behavior would be undefined ... but it would almost certainly crash as soon as the code tried to return to some address on the top of the corrupted stack or follow an indirect pointer from one part of the heap to another or ...

In any event you won't see it on any modern, general purpose workstation or server. You'll hit a resource limit and get malloc failures, or you'll run into virtual memory and eventually the system will thrash itself into quivering pile of "hit the red switch."