App crash after I call Detoured function I created @ Win7-x64

Question

I'm new in here and its my first question asked. I hope to get along in here just fine :) And for my question!:

I wrote a Detouring infrastructure for x64 processor (normal intel x64)

The main interesting functions looks like this:

typedef ULONG_PTR (WINAPI *MESSAGEBOX)(HWND, LPCSTR, LPCSTR, UINT);

    DWORD WINAPI DETOUR_MyMessageboxA(MESSAGEBOX OriginalFunction, HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType)
    {
        OutputDebugPrintA("Success.");
        return OriginalFunction(hWnd, lpText, lpCaption, uType);
    }

When I detour MessageBoxA, I wrote the following ASM code:

push rdi
push r9

sub rsp, 20
mov rdi, rsp

mov r9, r8
mov r8, rdx
mov rdx, ecx
mov ecx, [ORIGINAL_ADDRESS]

call DETOUR_MyMessageboxA

add rsp, 20

pop r9
pop rdi
ret

The idea is to basically "Shift right" the arguments and add another argument (My original function address) to the detour. I hope I got it right, the calling convention is as the following:

First Arg: RCX

Second Arg: RDX

Third Arg: R8

Fourth Arg: R9

Anything more in rsp (under the "Shadow space" thingy)

Just making sure in on the same page with everyone.. the stack should look like this before calling a function:

    GARBAGE.....
RSP RetAddress
    8BYTE-GARBAGE - [Shadow space]
    8BYTE-GARBAGE - [Shadow space]
    8BYTE-GARBAGE - [Shadow space]
    8BYTE-GARBAGE - [Shadow space]
    Arg5
    Arg6
    ...

Now after I begin me detour and my Assembly code is called instead of the original function I get an exception:

0:000> g
(624.17b4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
ntdll!RtlCaptureContext+0x86:
00000000`76d508c5 0fae8100010000  fxsave  [rcx+100h]    ds:00000000`0026ead8=50
0:000> k
Child-SP          RetAddr           Call Site
00000000`0026e998 00000000`76d195c8 ntdll!RtlCaptureContext+0x86
00000000`0026e9a8 000007fe`fccc940d ntdll!RtlRaiseException+0x48
00000000`0026efe8 000007fe`fccdaa0d KERNELBASE!RaiseException+0x39
00000000`0026f0b8 000007fe`f6ec2c7a KERNELBASE!OutputDebugStringA+0x6d
00000000`0026f388 00000000`0014002f Detour_RekSai!DETOUR_MyMessageboxA+0x3a [c:\projects\test.c @ 14]
00000000`0026f3b8 00000000`00130000 0x14002f
00000000`0026f3c0 00000000`00000000 0x130000
0:000> !vprot 00000000`0026ead8
BaseAddress:       000000000026e000
AllocationBase:    00000000001f0000
AllocationProtect: 00000004  PAGE_READWRITE
RegionSize:        0000000000002000
State:             00001000  MEM_COMMIT
Protect:           00000004  PAGE_READWRITE
Type:              00020000  MEM_PRIVATE
0:000> db rcx+100 L20
00000000`0026ead8  50 f0 26 00 00 00 00 00-58 f9 26 00 00 00 00 00  P.&.....X.&.....
00000000`0026eae8  00 00 00 00 00 00 00 00-e9 c9 ec f6 fe 07 00 00  ................
0:000> !gle
LastErrorValue: (NTSTATUS) 0xc0000008 (3221225480) - An invalid HANDLE was specified.
LastStatusValue: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

Now I have no idea what is causing this.. I think I got all the variables and stuck correctly.. didn't I? I have no clue what this memory location is or why it happens.

Thanks for any help..

EDIT: If I remove the "OutputDebugPrint" in my function:

DWORD WINAPI DETOUR_MyMessageboxA(MESSAGEBOX OriginalFunction, HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType)
{
    return OriginalFunction(hWnd, lpText, lpCaption, uType);
}

I still get an exception (A different one, but Something tells me they are both caused because of the same thing).

0:000> g
(1e3c.298c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
LPK!LpkCharsetDraw+0x78:
000007fe`fe811775 440f29842450010000 movaps xmmword ptr [rsp+150h],xmm8 ss:00000000`0026ebf8=000007fefe91a8e40000000002051320
0:000> k
Child-SP          RetAddr           Call Site
00000000`0026eaa8 000007fe`fe8114cc LPK!LpkCharsetDraw+0x78
00000000`0026ec68 00000000`76af85c5 LPK!LpkDrawTextEx+0x68
00000000`0026ecd8 00000000`76af87fc USER32!DT_DrawStr+0xa6
00000000`0026ed78 00000000`76af8216 USER32!DT_GetLineBreak+0x241
00000000`0026ee28 00000000`76b51e07 USER32!DrawTextExWorker+0x3f8
00000000`0026ef38 00000000`76b523e9 USER32!MB_CalcDialogSize+0x187
00000000`0026efd8 00000000`76b51c15 USER32!SoftModalMessageBox+0x47d
00000000`0026f108 00000000`76b5146b USER32!MessageBoxWorker+0x31d
00000000`0026f2c8 00000000`76b51362 USER32!MessageBoxTimeoutW+0xb3
00000000`0026f398 000007fe`ecfe2c85 USER32!MessageBoxW+0x4e
00000000`0026f3d8 00000000`0015002f Detour_RekSai!DETOUR_MyMessageboxA+0x45 [c:\users\ariel\documents\visual studio 2012\projects\detour_reksai\detour_reksai\test.c @ 19]
00000000`0026f408 00000000`00140000 0x15002f
00000000`0026f410 00000000`00000000 0x140000
0:000> !vprot [rsp+150h]
BaseAddress:       000000000026e000
AllocationBase:    00000000001f0000
AllocationProtect: 00000004  PAGE_READWRITE
RegionSize:        0000000000002000
State:             00001000  MEM_COMMIT
Protect:           00000004  PAGE_READWRITE
Type:              00020000  MEM_PRIVATE
0:000> db [rsp+150h] L20
00000000`0026ebf8  20 13 05 02 00 00 00 00-e4 a8 91 fe fe 07 00 00   ...............
00000000`0026ec08  01 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
0:000> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

Answer 1

You might have missed this line: This exception may be expected and handled. The exception as reported is coming from the OutputDebugPrintA, and I doubt that has much to do with your detouring logic. Make sure this is a real unhandled exception.

PS: Not sure what you wanted to do with mov edi, esp.

---

As per the comments, it seems your problem is rsp misalignment. You should maintain 16 byte alignment. In this case, you are pushing 2 items (worth 16 bytes) and then adjusting by 32, and finally you have a call which places 8 bytes of return address on the stack. All in all, you have 8 byte misalignment. Since in this code you don't need rdi, you can simply omit the push/pop for that and thereby fix the misalignment. Also, using mov makes the situation clearer:

sub rsp, 28h

mov [rsp+20h], r9
mov r9, r8
mov r8, rdx
mov rdx, rcx
mov ecx, [ORIGINAL_ADDRESS]

call DETOUR_MyMessageboxA

add rsp, 28h
ret

---

How I should implement a lets say 5 variable function detour?

mov rax, [rsp+28h] ; 5th arg
sub rsp, 38h       ; space for 6 args + alignment

mov [rsp+28h], rax
mov [rsp+20h], r9
mov r9, r8
mov r8, rdx
mov rdx, rcx
mov ecx, [ORIGINAL_ADDRESS]

call DETOUR

add rsp, 38h
ret