Port Win32 DLL hook to Linux

Question

I have a program (NWShader) which hooks into a second program's OpenGL calls (NWN) to do post-processing effects and whatnot.

NWShader was originally built for Windows, generally modern versions (win32), and uses both DLL exports (to get Windows to load it and grab some OpenGL functions) and Detours (to hook into other functions). I'm using the trick where Win will look in the current directory for any DLLs before checking the sysdir, so it loads mine. I have on DLL that redirects with this method:

#pragma comment(linker, "/export:oldFunc=nwshader.newFunc)

To send them to a different named function in my own DLL. I then do any processing and call the original function from the system DLL.

I need to port NWShader to Linux (NWN exists in both flavors). As far as I can tell, what I need to make is a shared library (.so file). If this is preloaded before the NWN executable (I found a shell script to handle this), my functions will be called. The only problem is I need to call the original function (I would use various DLL dynamic loading methods for this, I think) and need to be able to do Detour-like hooking of internal functions.

At the moment I'm building on Ubuntu 9.10 x64 (with the 32-bit compiler flags). I haven't been able to find much on Google to help with this, but I don't know exactly what the *nix community refers to it as. I can code C++, but I'm more used to Windows. Being OpenGL, the only part the needs modified to be compatible with Linux is the hooking code and the calls. Is there a simple and easy way to do this, or will it involve recreating Detours and dynamically loading the original function addresses?

Answer 1

The library preloading can be done through LD_PRELOAD. From there you want to look at the dlopen and dlsym calls to get at the actual functions in the original library. This is all if you want to do things by hand.

You can also look at modifying ltrace in a way such that you provide the functions to hook (via the -e flag) and let ltrace handle the bookkeeping for you.

[Edit] An example of doing this by hand:

#include <dlfcn.h>
#include <stdio.h>

int (*orig_puts)(const char *);

int puts (const char * str) {
    void * handle = dlopen("/lib/libc.so.6", RTLD_NOW | RTLD_GLOBAL);
    orig_puts = dlsym(handle,"puts");
    fprintf (stderr,"I have hooked your puts\n");
    return orig_puts(str);
}

And with a program such as

#include <stdio.h>

int main () {
    puts ("Hello World");
    return 0;
}

You get the following:

$ ./a.out
Hello World
$ LD_PRELOAD=./libhook ./a.out
I have hooked your puts
Hello World