Shadow space example

Question

EDIT:

I have accepted an answer below and also added my own with my final revision of the code. Hopefully it shows people actual examples of Shadow Space allocation rather than more words.

EDIT 2: I also managed to find a link to a calling conventions PDF in the Annotations of a YouTube video (of all things) which has some interesting tidbits on Shadow Space and the Red Zone on Linux. It can be found here: http://www.agner.org/optimize/calling_conventions.pdf

ORIGINAL:

I have looked at a couple of other questions here and all over the internet but I can't seem to find a proper example of allocating "Shadow Space" when calling a subroutine/Windows API in 64 bit Windows assembly.

My understanding is this:

• Caller should sub rsp,<bytes here> prior to call callee
• Callee should use it to store registers if need be (or local variables, if register saving isn't required)
• Caller cleans it up, e.g: add rsp,<bytes here>
• The amount allocated should be aligned to 32 bytes

With that in mind, this is what I have tried:

section .text

start:

    sub rsp,0x20 ; <---- Allocate 32 bytes of "Shadow space"

    mov rcx,msg1
    mov rdx,msg1.len
    call write

    add rsp,0x20

    mov rcx,NULL
    call ExitProcess

    ret

write:

    mov [rsp+0x08],rcx      ; <-- use the Shadow space
    mov [rsp+0x10],rdx      ; <-- and again

    mov rcx,STD_OUTPUT_HANDLE   ; Get handle to StdOut
    call GetStdHandle

    mov rcx,rax         ; hConsoleOutput
    mov rdx,[rsp+0x08]      ; lpBuffer
    mov r8,[rsp+0x10]       ; nNumberOfCharsToWrite
    mov r9,empty        ; lpNumberOfCharsWritten
    push NULL           ; lpReserved
    call WriteConsoleA

    ret

My two strings are "Hello " and "World!\n". This manages to print "Hello " before crashing. I have a suspicion that I am doing it correctly ... except I should be cleaning up somehow (and I'm not sure how).

What am I doing wrong? I have tried a combination of sizes and also tried "allocating Shadow Space" prior to the WinAPI calls too (am I supposed to be doing that?).

It should be noted that this works perfectly fine when I don't care about Shadow Space at all. However, I am trying to be compliant with the ABI since my write function calls WinAPIs (and is therefore, not a leaf function).

Answer 1

The shadow space must be provided directly previous to the call. Imagine the shadow space as a relic from the old stdcall/cdecl convention: For WriteFile you needed five pushes. The shadow space stands for the last four pushes (the first four arguments). Now you need four registers, the shadow space (just the space, contents don't matter) and one value on the stack after the shadow space (which is in fact the first push). Currently the return address to the caller (start) is in the space that WriteFile will use as shadow space -> crash.

You can create a new shadow space for the WinAPI functions (GetStdHandle and WriteConsoleA) inside the function write:

write:
    push rbp
    mov rbp, rsp
    sub rsp, (16 + 32)      ; 5th argument of WriteConsoleA (8) + Shadow space (32)
                            ; plus another 8 to make it a multiple of 16 (to keep stack aligned after one push aligned it after function entry)

    mov [rbp+16],rcx        ; <-- use our Shadow space, provided by `start`
    mov [rbp+24],rdx        ; <-- and again, to save our incoming args

    mov rcx, -11            ; Get handle to StdOut
    call GetStdHandle

    mov rcx,rax             ; hConsoleOutput
    mov rdx, [rbp+16]       ; lpBuffer        ; reloaded saved copy of register arg
    mov r8, [rbp+24]        ; nNumberOfCharsToWrite
    mov r9,empty            ; lpNumberOfCharsWritten
    mov qword [rsp+32],0    ; lpReserved - 5th argument directly behind the shadow space
    call WriteConsoleA

    leave
    ret