Hooking (hot patching) class member functions. Modifying vtable entries

Question

I have 2 questions, after a rather long preamble.

By looking at any function pointer as at a void* I am able to modify its first instructions, transform them into a jmp (either 32 bit relative or 64 bit absolute, via r11, depending on x86/x86-64). I believe that looking at function code as at data is illegal in both C and C++, but it somehow seems to work, in an unsupported way, in both MSVC (Win32) and GCC (OS X). There are several places on the Internet saying that casting function pointers to void* is illegal.

One doesn't simply get a pointer to a class member. I mean, the compiler directly throws errors at build time when trying to look at such a pointer in the same way as I'd look at a void *, practice which seems to work just fine for non-member functions.

Fortunately, in order to hook Direct3D9, I'm working with stuff like IDirect3DDevice9 which has a vtable. With pDev of type IDirect3DDevice9*, it's enough that I look at pDev as at a PVOID*. Then, the first value at pDev is the address of an array of pointers to functions (the vtable):

// IDirect3DDevice9::Present()
typedef HRESULT (CALLBACK *PRESENT_PROC)(
    LPDIRECT3DDEVICE9, const RECT*,
    const RECT*,
    HWND,
    const RGNDATA*
);

PVOID (*vPtr)[] = reinterpret_cast<PVOID (*)[]>(
    *reinterpret_cast<PVOID*>(pDev)
);
PRESENT_PROC pDevicePresent = reinterpret_cast<PRESENT_PROC>(
    (*vPtr)[17]
);

since Present is the 18th entry.

The first answer from here gives a more elegant, higher-level method, starting with defining CINTERFACE. I haven't tested it yet, but according to it I can do things like

reinterpret_cast<PVOID>(pDev->lpVtbl->Present)

without error.

First problem. I'm not an awesome C++ programmer; how do I get a pointer, in general, to a member function, so that I can overwrite the executable bytes of that function. For a non-member I do:

#include <windows.h>
#include <cstdio>
using namespace std;

const unsigned char OP_JMP = 0xE9;  // 32 bit relative jmp
const SIZE_T SIZE_PATCH = 5;        // jmp dword ptr distance; 1 byte + 4 bytes
typedef void (*MyProc)();

void SimpleFunction1()
{
    printf("foo\n");
}

void SimpleFunction2()
{
    printf("bar\n");
}

int main()
{
    PBYTE foo = reinterpret_cast<PBYTE>(SimpleFunction1);
    PBYTE bar = reinterpret_cast<PBYTE>(SimpleFunction2);

    DWORD oldProtection;
    // make sure the bytes of the function are writable
    // by default they are only readable and executable
    BOOL res = VirtualProtect(
        foo,
        SIZE_PATCH,
        PAGE_EXECUTE_READWRITE,
        &oldProtection
    );
    if (!res) return 1;

    // be mindful of pointer arithmetic
    // works with PBYTE, won't with PDWORD
    DWORD distanceToNewFoo = bar - foo - SIZE_PATCH;

    *foo = OP_JMP;
    *reinterpret_cast<PDWORD>(foo + 1) = distanceToNewFoo;

    // called though the pointer instead of foo()
    // to make sure the compiler won't inline or do some other stupid stuff
    reinterpret_cast<MyProc>(foo)(); // will print "bar\n"
    return 0;
}

and something in the same vein for x86-64. For a virtual member of an object, I get the foo pointer from the vtable itself as I've shown above:

reinterpret_cast<FUNC_TYPE>(
    *(reinterpret_cast<void**>(
        *reinterpret_cast<void**>(objptr)) + n
    )
)

Second problem. Can't I get by with simply modifying the entries from the vtable for my object? Here's an example, needless to say, it doesn't work for the pDev object, as taken directly from Direct3D, but Taksi seems to use this method:

#include <cstdio>
using namespace std;

class BaseClass
{
public:
    BaseClass(int a = 0, int b = 0);
    int GetA();
    int GetB();
    virtual void Test();
private:
    int _a;
    int _b;
};

BaseClass::BaseClass(int a, int b) :
    _a(a),
    _b(b)
{
}

int BaseClass::GetA()
{
    return _a;
}

int BaseClass::GetB()
{
    return _b;
}

void BaseClass::Test()
{
    printf("test %d; %d\n", _a, _b);
}

void TheNewFunction(BaseClass *bc)
{
    printf("I am an intruder\n");
}

typedef void (*PROC_TYPE)(BaseClass *);

int main()
{
    BaseClass foo(5, 56);
    PROC_TYPE proc = 0;
    proc = reinterpret_cast<PROC_TYPE>(
        *reinterpret_cast<void**>(
            *reinterpret_cast<void**>(&foo)
        )
    );
    proc(&foo);
    reinterpret_cast<void**>(
        *reinterpret_cast<void**>(&foo)
    )[0] = reinterpret_cast<void*>(TheNewFunction);

    foo.Test(); // runs same old Test(); maybe due to compiler optimization?
    proc = reinterpret_cast<PROC_TYPE>(
        *reinterpret_cast<void**>(
            *reinterpret_cast<void**>(&foo)
        )
    );

    proc(&foo); // runs TheNewFunction
    BaseClass *goo = &foo;
    goo->Test(); // runs TheNewFunction
    return 0;
}

Answer 1

The fastest way to perform this ugly cast (member function to void *), would be the infamous union_cast<>:

template <class T1, class T2>
T1 union_cast(T2 v)
{
  static_assert(sizeof(T1) >= sizeof(T2), "Bad union_cast!");
  union UT {T1 t1; T2 t2;} u {};
  u.t2 = v;
  return u.t1;
}

Use like this:

class MyClass
{
public:
  void foo(int);
};

auto p = union_cast<void *>(&MyClass::foo);

Now, I've given you a loaded gun with the safety off. Please use with care...

Answer 2

As you pointed-out, your method is not "portable", but it actually works in your specific case. And there's no problem IMHO.

First answer:

This is the syntax for working with member function pointers:

class SomeClass {
    int SomeFunc(int, int);
};

int (SomeClass::* pfn)(int, int); // variable pfn is a pointer to a SomeClass's member function

pfn = &SomeClass::SomeFunc; // assign this variable to the member function with the adequate prototype.

SomeClass obj; // instance of this class

int res = (obj.*pfn)(1, 2); // call the member function pointer

Second answer:

You may modify the members of the vtable directly, but you should know that by this you subclass all the objects of this class (and possibly some derived classes).

If you want to subclass only one specific obj you should create another function table, and overwrite the object's vtable to point to the new function table.