Is it possible to call a non-exported function that resides in an exe?

Question

I'd like to call a function that resides in a 3rd-party .exe and obtain its result. It seems like there should be a way, as long as I know the function address, calling-convention, etc... but I don't know how.

Does anyone know how I would do this?

I realize that any solution would be a non-standard hack, but there must be a way!

---

My non-nefarious use-case: I'm reverse engineering a file-format for my software. The calculations in this function are too complex for my tiny brain to figure out; I've been able to pull the assembly-code directly into my own DLL for testing, but of course I can't release that, as that would be stealing. I will be assuming users already have this particular application pre-installed so my software will run.

Answer 1

It is possible but not trivial. And yes, this is a very dirty hack.

In some cases loading the EXE file with LoadLibrary is enough. The returned HMODULE is actually the base address of the loaded EXE. Cast it to a suitable int type, add your relative function address to that, cast it back to a function pointer and call the function through that pointer.

Unfortunately, the EXE file may have its relocation info stripped. It means that the EXE will be expecting to run from a specific address. In this case, you have to change your own program's base address to avoid conflict. Check out your linker's docs, there should be an option to do that. After that, LoadLibrary will load the EXE in its preferred base address and hopefully all should work fine.

There is some very useful info on this here. Make sure to check the update at the end of the page for a different technique that may work better in some cases.

Edit: As Alex correctly stated in the comment below, if the function relies on some initialized value, or it calls such a function, including most C runtime functions, it will be much harder to make it work. One can identify the initialization functions and call them beforehand but using debug API may be your best bet in those situations.