Render inside of OpenGL scene from an external application using an injected DLL

Question

I've been able to inject DLLs in to applications for things like keyboard actions, but how does one hook in to an OpenGL program and draw over it? I've read many "tutorials" and answers, but they all outline the idea behind doing it, not how to actually do it. If someone could point me in the right direction, I'd much appreciate it.

What I intend to do is be able to draw an overlay (or maybe draw into the actual scene? Is that possible?) from a 3rd party application in RAM only (injected dll). My only experience with OpenGL is via GLUT, but I'm willing to learn if need be.

After hours of research, I have found an open-source application that can do what I want, and looked through its code, but can't seem to find what method is being hooked or and how the ASM code is changed at that specific method.

Is the method to be hooked defined when the dll is injected, or does the dll do it upon being attached? From my understanding, you could do either, but is one better than the other?

I'm looking for a comprehensive tutorial, or a simple example of such a thing. If someone could point me in the right direction, that'd be great!

Answer 1

The most critical step to understand is, that OpenGL is a state based drawing API, not a scene graph. Hence you must not assume, that there is some scene you can add some overlay to. Instead you have to find a point, where you know the program you injected is done drawing and you can draw your overlay over it.

This very point is easily identified for double buffered windows: The call to wglSwapBuffers (or glXSwapBuffers on X11 based systems, eglSwapBuffers for EGL based systems – Windows uses neither of them).

So what you do is hooking into the call to SwapBuffers. You redirect that call into your overlay drawing code.

Since OpenGL is state based you should avoid messing with the context of the program you've hooked into. No problem, you can create as many OpenGL contexts as you like and use them all on the same drawable. So your hooked SwapBuffers would look something like this:

BOOL WINAPI (*host_SwapBuffers)(HDC);
HGLRC hook_last_rc_created = NULL; /* for context sharing */

/* C++ std::map for the lazy. */
std::map<int, HGLRC> hook_pixelformat_to_rc;

/* This function is called to setup per RC resources, i.e. things
 * that are not shared through the list namespace, like FBOs, VAOs
 * and such. */
void init_RC();

/* This function is called when the first hook OpenGL context is
 * created. This is to be used for one time initialization of OpenGL
 * objects that are shared among all contexts through the list namespace,
 * like textures, VBOs, PBOs and such. */
void init_GL();

/* The function that does the actual overlay drawing.
 * Must not call SwapBuffers when finished. The draw overlay has to set
 * up the viewport, projection and modelview transformations.
 *
 * rect gives the dimensions of the window we draw to, for
 * setting the viewport and stuff like that. */
void draw_overlay(RECT const *rect);

BOOL WINAPI hook_SwapBuffers(HDC dc)
{
    /* It's illegal to call SwapBuffers with a NULL dc. To keep Windows
     * error behavior we bail out, forwarding the call to the system
     * SwapBuffers if dc == NULL */
    if( !dc ) {
        goto terminate;
    }

    /* OpenGL RCs can be shared between DCs of compatible pixelformat. So technically
     * any number of pixelformat IDs could be compatible. But we just assume that
     * compatibility is only between pixelformats of the same ID, which is safe. */
    int const pixelformat = GetPixelFormat(dc);

    HGLRC hook_rc;
    /* we need to create a RC for our hook only one time and can reuse that
     * later. It can even be used for different DCs or windows, as long as
     * they have been created with compatible pixel formats. To make this
     * hook really robust there is to be a mapping from DCs to pixelformat IDs.
     * If no RC can be found for the given pixelformat, create one. */
    if( 0 == hook_pixelformat_to_rc.count(pixelformat) ) {
        /* the DC is of a window that already got everything set up for OpenGL.
         * All we have to do is create our own HRC if we don't have one already. */

        hook_rc = wglCreateContext(dc);
        if( !hook_rc ) {
            /* bail out, but forward the SwapBuffers call on error. Don't use
             * exceptions in hook DLLs, or very bad things may happen, if you're
             * not very, very careful. */
            goto terminate;
        }

        /* This could throw an exception... */
        hook_pixelformat_to_rc[pixelformat] = hook_rc;
        init_RC();

        /* If there was a RC created before, share its list namespace, so
         * that textures, VBOs, PBOs and so on are available among all the
         * contexts our hook uses. */
        if( hook_last_rc_created ) {
            wglShareLists(hook_last_rc_created, hook_rc);
        }
        else {
            init_GL();
        }
        hook_last_rc_created = hook_rc;
    }
    else {
        hook_rc = hook_pixelformat_to_rc[pixelformat];
    }

    /* Preserve the RC bound to the window before our hook.
     * May be NULL, but that's fine, this just means there was nothing
     * bound and we should return to that state at the end. */
    HGLRC const host_rc = wglGetCurrentContext();

    /* Find out which window the DC belongs to, so that we can query for its
     * dimension, which we need to properly setup a viewport region. */
    HWND const wnd = WindowFromDC(dc);
    RECT wnd_rect;
    GetClientRect(wnd, &wnd_rect);

    /* bind our own RC */
    wglMakeCurrent(dc, hook_rc);

    /* we can now draw the overlay */
    draw_overlay(&wnd_rect);

    /* Restore RC to DC binding */
    wglMakeCurrent(dc, host_rc);
terminate:
    /* host_SwapBuffers is a function pointer initialized to the address of the
     * system SwapBuffers function before hooking by the DLL injection code. */
    return host_SwapBuffers(hdc);
}