How to prevent a specific DLL from loading into my process

Question

I think I have researched this pretty thoroughly and I have not found an acceptable answer. First the broad strokes: Windows 8.1, Visual Studio 2013. Although, I don't think these are important.

Problem as follows. The application I am writing makes use of A.dll. We use a third-party vendor product (a backup program, but again this is not important) that has installed a Context Menu Handler control under HKEY_CLASSES_ROOT\Directory\shellex\ContextMenuHandlers. Let's say the path to this is c:\Program Files\Vendor\control.dll.

Now, the issue is that when my program opens a file chooser dialog (it's a Qt program that uses QFileDialog which then uses the standard Windows one), this control is loaded to provide context-sensitive right-click functionality. This control depends on a different version of "A.dll" and when control.dll is loaded, my program promptly crashes.

I don't need this extra functionality. What I would love to do is to prevent this specific dll (control.dll) from loading in my process. In an object-oriented world I would simply overload LoadLibrary(), check for this specific DLL, and then call the standard one otherwise. However this doesn't seem feasible.

Is there an easy way to do this?

Thanks! Dan

Answer 1

To prevent the vendor.dll from loading you can use a hook on the following Win32API function LoadLibrary and LoadLibraryEx which are responsible for dynamically loading DLLs and which are also used to load shell extensions. The hook is really assembler code at the code site of the LoadLibrary function, which redirects (jumps) to a function defined by yourself. In this function you can then intercept any call to vendor.dll being loaded and just return 0, which indicates that the library could not be loaded.

Some example code how to go about it using MinHook library:

HMODULE WINAPI LoadLibraryA_check(_In_ LPCTSTR lpFileName)
{
  if (isInWhiteList(lpFileName))
    return loadLibraryA_Original(lpFileName);
  else 
  {
    // Pretend that the module was not found by returning
    // 126 (0x7E): "The specified module could not be found."
    SetLastError(ERROR_MOD_NOT_FOUND);         
    return NULL;
  }
}

bool installWhitelistFilter()
{
  // Initialize MinHook.
  if (MH_Initialize() != MH_OK)
    return false;

  if (MH_CreateHook(&LoadLibraryA, &LoadLibraryA_check, 
      reinterpret_cast<LPVOID*>(&loadLibraryA_Original)) != MH_OK)
    return false;

  if (MH_EnableHook(&LoadLibraryA) != MH_OK)
    return false;

  // same for LoadLibraryW, LoadLibraryExW, LoadLibraryExA

  return true;
}