A x64-DLL injected into a x64-process hooking a x86-DLL fails using C++ and EasyHook

Question

A x64-DLL injected into a x64-process hooking a x86-DLL fails using C++ and EasyHook. It works if Loader, InjectionLibrary and InjectionTarget(it's available in both versions and i need both to be hooked) are x86. Getting the address of the exported procedure(GetProcAddress itself) isn't a problem at x64. The InjectionTarget has HookTarget(Kernel32.dll) as a dependency at x64 aswell. LhInstallHook(...) returns STATUS_NOT_SUPPORTED where the source comments say that happens when: "The target entry point contains unsupported instructions."

Due to the fact that the source is fine for x86 builds i've decided to not add it.

I've scratched a little diagram

Answer 1

You cannot use a 32-bit DLL in a 64bit process, and indeed, this generalizes- you cannot mix and match x86 and x64 code, a single process is either entirely x64, or entirely x86. That's fundamental to x86-64 and there's nothing you can do about it. In the Windows control console, they make a 64bit process and a 32bit process and use IPC to control the 32bit process to load and deal with all the 32bit shell extensions. You could try something similar, if you're brave and/or desperate.

Edit: Wait a minute, wait a minute. Could you describe this process a little more at basics when everything works fine in x86 mode? Like, X loads a function from Y, I am doing Z, because it appears that I don't understand what you're doing.

You have an injection target, and an injected DLL, from which you're trying to call procedures in the third-party x86-only DLL. So the normal flow of call goes from InjectionTarget -> InjectionLibrary -> HookTarget -> ExportedProcedure. And this isn't working for you because HookTarget is x86 only and you can't change that, so when you recompile InjectionLibrary for x64 for the x64 version of InjectionTarget, it doesn't work anymore because your x64 InjectionLibrary is trying to load an x86 HookTarget.

The only solution to this problem is to create an x86 process and use inter-process communication to get it to call the procedures in HookTarget that you want called. If you can't re-compile HookTarget for x64, then this is the only way to do it.