How to intercept API method calls in a 64bit process?

Question

Background

I'm working on a legacy product that can successfully, via an injection dll, intercept an arbitrary method call that an injectee process is trying to make into an arbitrary dll. In particular, the gdi32.dll library. Unfortunately, it does not work when it is embedded in 64bit applications. Its become a hot button topic, and it is time to upgrade its functionality. Also unfortunately, the source is barren of comments (typical >:-<), and from the looks of it, whoever wrote this was fairly familiar with the x86 instruction set. I haven't worked with assembly in years, and when I did it was Motorola assembly.

After scouring the the internet I came across this article from an Intel employee. If our source code didn't pre-date this article by about 7 years, I'd say that this is exactly where our Mr. NoComments developer learned to perform API method interception. That's how similar the procedure is. This article is also summed up in a nice pdf (Intercepting System API calls), which can also be found linked from the aforementioned website.

Problem

I'd like to to really understand the example provided in the Intel webpage link so that I may take a good crack at creating a solution for a 64bit scenario. It's well documented and is a little easier for me to understand. Below is an excerpt with the InterceptAPI() routine. I've added my own comments indicated by "//#" (original comments are noted by the standard "//"), where I explain what I think I know and what I don't know:

BOOL InterceptAPI(HMODULE hLocalModule, const char* c_szDllName,
    const char* c_szApiName, DWORD dwReplaced, DWORD dwTrampoline, int offset) 
{ 
    //# Just a foreword.  One of the bigger mysteries of this routine to me is
    //# this magical number 5 and the offset variable.  Now I'm assuming, that
    //# there are 5 bytes at the beginning of every method that are basically 
    //# there to set up some sort of pre-method-jump context switch, since its
    //# about to leave the current method and jump to another.  So I'm guessing
    //# that for all scenarios, the minimum number of bytes is 5, but for some
    //# there may be more than 5 bytes so that's what the "offset" variable is
    //# for. In the aforementioned article, the author writes "One additional 
    //# complication exists, in that the sixth byte of the original code may be
    //# part of the previous instruction. In that case, the function overwrites
    //# part of the previous instruction and then crashes."  So some method
    //# starting code contains multi-byte opcodes while others don't apparently.
    //# And if you don't know the instruction set well enough, I'm guessing
    //# you'll just have to figure it out by trial and error.
    int i; 
    DWORD dwOldProtect;

    //# Fetching the address of the method that we want to capture and reroute
    //# Example: c_szDllName="user32",   c_szApiName="SelectObject"
    DWORD dwAddressToIntercept = (DWORD)GetProcAddress( 
        GetModuleHandle((char*)c_szDllName), (char*)c_szApiName); 


    //# Storing address of method we are about to intercept in another variable
    BYTE *pbTargetCode = (BYTE *) dwAddressToIntercept;

    //# Storing address of method we are going to use to take the place of the 
    //# intercepted method in another variable.
    BYTE *pbReplaced = (BYTE *) dwReplaced; 

    //# "Trampoline" appears to be a "Microsoft Detours" term, but its basically
    //# a pointer so that we can get to the original "implementation" of the method
    //# we are intercepting.  Most of the time your replacement function will
    //# want to call the original function so this is pretty important.  What its
    //# pointing to must already be pre allocated by the caller.  The author of
    //# the aforementioned article states "Prepare a dummy function that has the
    //# same declaration that will be used as the trampoline. Make sure the dummy
    //# function is more than 10 bytes long." I believe I'd prefer allocating this
    //# memory within this function itself just to make using this InterceptAPI()
    //# method easier, but this is the implementation as it stands.
    BYTE *pbTrampoline = (BYTE *) dwTrampoline; 


    // Change the protection of the trampoline region 
    // so that we can overwrite the first 5 + offset bytes.
    //# This is voodoo magic to me, but I'm guessing you just can't hop on the
    //# stack and start changing execute instructions without ringing some
    //# alarms, so this makes sure the alarms don't ring. Here we are allowing
    //# permissions so we can change the bytes at the beginning of our
    //# trampoline method.
    VirtualProtect((void *) dwTrampoline, 5+offset, PAGE_WRITECOPY, &dwOldProtect); 

    //# More voodoo magic to me, but this appears to be a way to copy over extra
    //# opcodes that may be needed.  Some opcodes are multi byte I believe so this
    //# is where you can make sure you don't miss them.
    for (i=0;i<offset;i++) 
        *pbTrampoline++ = *pbTargetCode++; 

    //# Resetting the pbTargetCode pointer since it was modified it in the above
    //# for loop.
    pbTargetCode = (BYTE *) dwAddressToIntercept; 


    // Insert unconditional jump in the trampoline.
    //# This is pretty understandable.  0xE9 the x86 JMP command.  I looked
    //# this up in Intel's documentation and it can be followed by a 16-bit
    //# offset or a 32-bit offset. The 16-bit version is not supported in 64-bit
    //# architecture but lets just hope they are all 32-bit and that this does
    //# indeed do what it is intended in 64-bit scenarios
    *pbTrampoline++ = 0xE9;        // jump rel32 

    //# So basically here it looks like we are following up our jump command with
    //# the address its supposed to jump too.  This is a relative offset, that's why
    //# we are subtracting pbTargetCode and pbTrampoline.  Also, since JMP opcodes
    //# jump relative to the address AFTER the jump address, that's why we are
    //# adding 4 to pbTrampoline.  Also, offset is added to pbTargetCode because we
    //# advanced the pointers in the for loop above an "offset" number of bytes.
    *((signed int *)(pbTrampoline)) = (pbTargetCode+offset) - (pbTrampoline + 4); 

    //# Not quite sure why we are changing the permissions on the trampoline function
    //# again, but looks like we are making it executable here.  Maybe this is the
    //# last thing we have to do before it is actually callable and usable.
    VirtualProtect((void *) dwTrampoline, 5+offset, PAGE_EXECUTE, &dwOldProtect); 


    // Overwrite the first 5 bytes of the target function 
    //# It seems we are now setting permissions so we can modify the original
    //# intercepted routine.  It is still pointing to its original code so we
    //# need to eventually redirect it.
    VirtualProtect((void *) dwAddressToIntercept, 5, PAGE_WRITECOPY, &dwOldProtect); 

    //# This will now instruct the original method to instead jump to the next
    //# address it sees on the stack.
    *pbTargetCode++ = 0xE9;        // jump rel32

    //# this is the address we want our original intercepted method to jump to.
    //# Where its jumping to will have the code of our replacement method.
    //# The "+ 4" is because the jump occurs relative to the address of the
    //# NEXT instruction after the 4byte address.
    *((signed int *)(pbTargetCode)) = pbReplaced - (pbTargetCode +4); 

    //# Changing the permissions of our original intercepted routine back to execute
    //# permissions so it can be called by other methods.
    VirtualProtect((void *) dwAddressToIntercept, 5, PAGE_EXECUTE, &dwOldProtect); 


    // Flush the instruction cache to make sure  
    // the modified code is executed.
    //# I guess this is just to make sure that if any instructions from the old
    //# state of the methods we changed, have wound up in cache, that it gets
    //# purged out of there before it gets used.
    FlushInstructionCache(GetCurrentProcess(), NULL, NULL); 

    return TRUE; 
} 

I think I have a pretty good understanding of whats going on here in this code. So the million dollar question is: What about this doesn't work for 64bit processes? My first thought was, "Oh, well the addresses should now be 8 bytes, so that's got to be whats wrong." But I think that the JMP command still only takes a relative 32bit address, so that op code should still be valid even with 32bit address in a 64bit process. Other than that the only thing i believe it could be is that our magical 5 bytes at the beginning of a method call is actually some other magical number. Anyone got some better insight?

Note: I know there are some other solutions out there like "Microsoft Detours" and "EasyHook". The former is too expensive and I'm currently exploring the latter, but its disappointing so far. So, I'd like to keep the discussion to this topic specifically. I find it interesting as well as the best solution to my issue. So please no "Hey, I don't know anything about this post in particular but try {insert 3rd party solution here} instead."

Answer 1

Since the code suggested looks to be target for Microsoft platforms, I would suggest you just use Detours. Using Detours your trampoline will work on 32 and 64 bit systems.