Is there anything that prevents stack overflow in assembly?

Question

In assembly language programming, what (if anything) prevents a stack from growing until it clobbers data or instructions?

Answer 1

Only the CPU/MMU configuration, which is usually done by the OS, can prevent or intercept stack overflows and memory corruptions or attempts to access something without the necessary privileges.

You can read the chapters on memory management in the x86 CPU manuals from Intel/AMD to find out more.

Answer 2

You can use canaries directly below the accepted lower bound of stack and check the canaries' values constantly.

In kernelland code (ring 0) you can also set a hardware breakpoint by setting the value of one of the debug registers dr0, dr1, dr2 and dr3 to the linear address of the breakpoint address, directly below the accepted lower bound of stack and then setting the correct flag bit of dr7 register, and report or try to fix the situation in the breakpoint handler code. See HardWare BreakPoints The Definitive Guide.