How to unload C++ Shell Extension DLL properly

Question

I have shell extension dll written in C++ and COM The dll is registered and loaded into memory. My upgrade setup program will do the following:

• Unregister the shell extension dll, kill the explorer.exe
• Copy the later version of the shell extension dll (Step 2)
• Start t he explorer.exe

It works fine. But the issue is:

If the user opened any other applications ( Internet explorer, sometimes windows task manager, notepad etc), the step Step 2 is failing.

Is there any way to close all shell extension dll hooks while upgrading the dll.

In the dll I am using GetOverlayInfo, context menu, database connection etc

Answer 1

Simply put: don't do it (i.e. do not unload it forcibly). You would have to enumerate all processes that have your shell extension loaded and then "restart" those. This is very invasive and quite frankly: bad behavior (for an installer). This also requires particular privileges which your installer may not otherwise need.

What most people still don't seem to be aware of is that MoveFile (and MoveFileEx) can be used to move DLLs or EXE files that are currently in use by a running application.

This is the approach Windows Installer takes. Have you ever noticed the folder \Config.msi in the root of a given drive after installing some .msi? This folder actually contains (moved and usually renamed to some unique "temporary" name) the original files that have been moved out but were still in use at the time. They are then usually scheduled for deletion upon boot (MoveFileEx with MOVEFILE_DELAY_UNTIL_REBOOT).

You can use the same mechanism in your homebrew installer and move the old file that is in use out of its original location and move the other one in right away. Any new application instance will then make use of the new shell extension (*), while the old one will continue to be used by any of the running applications that loaded it at one point or another.

(*) I'm not 100% sure about this because of the rules that apply to DLL loading and prevent modules with the same name to be loaded again under some circumstances.

Answer 2

Depending on your installer, you may be able to make use of the Restart Manager support in Windows Vista+. This will allow your setup to query for every application that is using your DLL, and try and gracefully shut them down. If they can't be, then you will need to register it for replacement on a restart. Once the setup has finished, the Restart Manager will try and restart any programs it shut down (that support being restarted).