Why does COM CoInitializeSecurity fail in my DLL?

Question

I'm currently studying VSHADOW.EXE 3.0 from the MS Windows SDK 6.1. I have made a version which can be compiled into a DLL that only exports one newly written function which expects the commandline as a string, tokenizes it, and then calls the old wmain. The DLL is not a COM server.

It works exactly as the old one when compiled as an EXE but doesn't quite work when compiled as a DLL because this call fails:

 CoInitializeSecurity(NULL, -1, NULL, NULL, 
                      RPC_C_AUTHN_LEVEL_PKT_PRIVACY, 
                      RPC_C_IMP_LEVEL_IDENTIFY, 
                      NULL, EOAC_NONE, NULL);

which fails with HRESULT error 0x80010119 (RPC_E_TOO_LATE, Security must be initialized before any interfaces are marshalled or unmarshalled. It cannot be changed once initialized.)

I run the exported function from a VB6 program where the function is imported with Declare Function vss Lib vshadow.dll ....

Does the error mean that the VB6 program already called CoInitializeSecurity? What can I do against the error?

Also, I have another question: why were exactly the security values RPC_C_AUTHN_LEVEL_PKT_PRIVACY and RPC_C_IMP_LEVEL_IDENTIFY chosen? What impact would other settings have?

Answer 1

There are a couple of standard COM calls that do not belong in a DLL. Like CoInitializeEx(), the call that initializes COM for a thread. The DLL doesn't own the thread, it is powerless to override the apartment state that the EXE selected.

CoInitializeSecurity() is another one, it is the job of the EXE to call it. Only it knows the proper values to pass, it's the one that determines the security policy. A DLL can't, it doesn't know anything about the client process.