What does "aligning the stack" mean in assembly?

Question

How does stack alignment work in ASMx64? When do you need to align the stack before a function call and how much do you need to subtract?

I didn't understand what was the purpose of it. I know there are other posts about this but it wasn't clear enough for me. For example:

extern foo
global bar

section .text
bar:
  ;some code...
  sub  rsp, 8     ; Why 8 (I saw this on some posts) ? Can it be another value ? Why do we need to substract?
  call foo        ; Do we need to align stack everytime we call a function?
  add  rsp, 8
  ;some code...
  ret

Answer 1

When do you need to align the stack before a function call and ....?

You need to align the stack when the function you're calling expects an aligned stack.

Functions that were written in other languages (e.g. C), and functions that are written in assembly but are designed to be called from other languages, will comply with some kind of calling convention (which includes much more than just stack alignment - how parameters are passed, where parameters are, things like "red zone", etc); and for 64-bit 80x86 the 2 common calling convention expect the stack to be aligned to a 16-byte boundary.

In a "pure assembly" project where you're calling functions that were written in assembly for assembly callers; the programmer is free to do whatever they like (e.g. whatever is best for performance) without caring about the limitations/restrictions of other languages that reduce performance (calling conventions). In this case you may never need to align the stack at all (but if you're dealing with AVX-512 a function might want the stack aligned to 64 bytes, and if you're dealing with AVX2 a function might want the stack aligned to 32 bytes, and ..).

... and how much do you need to substract?

If you don't know if the stack was aligned enough; then aligning the stack is typically done with AND (e.g. maybe and rsp,0xFFFFFFFFFFFFFFF0 to align the stack to a 16-byte boundary). This also means that you need to store the old stack pointer somewhere so that you can restore it; which often means 4 more instructions (push rbp, mov rbp,rsp before the alignment, then mov rsp,rbp and pop rbp to restore things later).

However; if you know that your caller aligned the stack for you (and that functions you call want the same or less alignment), then you can calculate how much extra to subtract by keeping track of how much you pushed on the stack. For example, if the stack was aligned to 32 bytes by your caller, and you push four 64-bit (8 byte) values on the stack and a call instruction will push another 64-bit value (return address); then it'd be a total of 5*8 = 40 bytes; so you'd know you need to subtract another 8 bytes to make the total 48 bytes if a you want to align to 16 bytes, or subtract another 24 bytes to make the total 64 bytes if you want to align to 32 bytes. This also avoids the need to save the original stack pointer (you can add whatever you subtracted later) so it can save 4 instructions.

Of course (for "pure assembly") you'd look at the requirements of all the functions you call and pick the worst case and align the stack to that once (and avoid aligning the stack multiple times differently, once for each function you call); and you might say "my function requires the stack to be aligned to whatever the worst case is for the functions I call" to ensure that you can calculate how much to subtract (and avoid the more expensive "AND with ..." approach). However (for "pure assembly") this places the burden on your caller (who may place the burden on their caller, who may....) so it can make performance worse (all of the ancestors in the call chain have to do extra work so you can avoid less work). In other words; for "pure assembly"; achieving the highest efficiency/performance takes a lot of work (to determine if/when stack should be aligned by how much and minimize the expense of ensuring stack is aligned where necessary).

This is also part of why compilers put the alignment in their calling conventions - a required "unlikely to be optimal most of the time" standard alignment makes it easier for the compiler.