Would having the call stack grow upward make buffer overruns safer?

Question

Each thread has its own stack to store local variables. But stacks are also used to store return addresses when calling a function.

In x86 assembly, esp points to the most-recently allocated end of the stack. Today, most CPUs have stack grow negatively. This behavior enables arbitrary code execution by overflowing the buffer and overwriting the saved return address. If the stack was to grow positively, such attacks would not be feasible.

Is it safer to have the call stack grow upwards? Why did Intel design 8086 with the stack growing downward? Could they have changed things in any later CPUs to let modern x86 have stacks that grow upwards?

Answer 1

Interesting point; most buffer overruns do go past the end, not before the beginning, so this would almost certainly help. Compilers could put local arrays at the highest address in a stack frame, so there wouldn't be any scalar locals to overwrite located after an array.

There's still danger if you pass the address of a local array to another function, though. Because the return address of the called function would be located just past the end of the array.

unsafe() {
    char buf[128];
    gets(buf);      // stack grows upward: exploit happens when gets executes `ret`
    // stack grows down: exploit happens when the `ret` at the end of *this* function executes.
}

So probably a lot of buffer overruns will still be possible. This idea only defeats buffer overruns when the unsafe array-writing code is inlined, so the overrun happens with nothing important above the array.

However, some other common causes of buffer overruns can easily be inlined, like strcat. Upward growing stacks will help sometimes.

Security measures don't have to be foolproof to be useful, so this would definitely help sometimes. Probably not enough for anyone to want to change an existing architecture like x86, but an interesting idea for new architectures. Stack-grows-down is a nearly universal standard in CPUs, though. Does anything use an upward-growing call stack? How much software actually depends on that assumption? Hopefully not much...

---

The traditional layout left room for the heap and/or the stack to grow, only causing a problem if they meet in the middle.

Predictable code/data addresses are more important than predictable stack addresses, so a computer with more RAM could put the stack farther from data/code, while still loading code/data at a constant address. (This is very hand-wavy. I consider myself lucky not to have written actual 16-bit programs, and only learned about but not used segmentation. Perhaps someone that still remembers DOS can shed some light here on why it works well to have the stack at a high address, instead of an upward-growing stack at the bottom of your segment and data/code at the top. e.g. with a "tiny" code model where everything is in one segment).

---

The only real chance to change this behaviour was with AMD64, which is the first time x86 has ever really broken backwards compatibility. Modern Intel CPUs still support 8086 undocumented opcodes like D6: SALC (Set AL from Carry Flag), limiting the coding space for ISA extensions. (e.g. SSSE3 and SSE4 instructions would be 1 byte shorter if Intel dropped support for undocumented opcodes.

Even then, it would only be for the new mode; AMD64 CPUs still have to support legacy mode, and when in 64-bit mode they have to mix long mode with compat mode (usually to run 32-bit user-space processes from 32-bit binaries).

AMD64 could maybe have added a stack-direction flag, but that would have made the hardware more complex. As I argued above, I don't think it would have been a big benefit for security. Otherwise, perhaps AMD architects would have considered it, but still unlikely. They were definitely aiming for minimally intrusive, and weren't sure it would catch on. They didn't want to be stuck with extra baggage to maintain AMD64 compatibility in their CPUs if the world mostly just kept running 32-bit OSes and 32-bit code.

That's a shame, because there are a lot of minor things they could have done that would probably not have required too many extra transistors in the execution units. (e.g. in long mode, replace setcc r/m8 with setcc r/m32).