How to avoid memory leaks using ShellExecuteEx?

Question

Minimal, Complete, and Verifiable example:

Visual Studio 2017 Pro 15.9.3 Windows 10 "1803" (17134.441) x64 Environment variable OANOCACHE set to 1. Data/Screenshots shown for a 32 bits Unicode build.

UPDATE: Exact same behavior on another machine with Windows 10 "1803" (17134.407) UPDATE: ZERO leaks on an old Laptop with Windows Seven UPDATE: Exact same behavior (leaks) on another machine with W10 "1803" (17134.335)

#include <windows.h>
#include <cstdio>

int main() {

    getchar();
    CoInitializeEx( NULL, COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE );
    printf( "Launching and terminating processes...\n" );
    for ( size_t i = 0; i < 64; ++i ) {

        SHELLEXECUTEINFO sei;
        memset( &sei, 0, sizeof( sei ) );
        sei.cbSize = sizeof( sei );
        sei.lpFile = L"iexplore.exe";
        sei.lpParameters = L"about:blank";
        sei.fMask = SEE_MASK_FLAG_NO_UI | SEE_MASK_NOCLOSEPROCESS | SEE_MASK_NOASYNC;
        BOOL bSuccess = ShellExecuteEx( &sei );
        if ( bSuccess == FALSE ) {
            printf( "\nShellExecuteEx failed with Win32 code %d and hInstApp %d. Exiting...\n",
                    GetLastError(), (int)sei.hInstApp );
            CoUninitialize();
            return 0;
        } // endif
        printf( "%d", (int)GetProcessId( sei.hProcess ) );
        Sleep( 1000 );
        bSuccess = TerminateProcess( sei.hProcess, 0 );
        if ( bSuccess == FALSE ) {
            printf( "\nTerminateProcess failed with Win32 code %d. Exiting...\n",
                    GetLastError() );
            CloseHandle( sei.hProcess );
            CoUninitialize();
            return 0;
        } // endif
        DWORD dwRetCode = WaitForSingleObject( sei.hProcess, 5000 );
        if ( dwRetCode != WAIT_OBJECT_0 ) {
            printf( "\nWaitForSingleObject failed with code %x. Exiting...\n",
                    dwRetCode );
            CloseHandle( sei.hProcess );
            CoUninitialize();
            return 0;
        } // endif
        CloseHandle( sei.hProcess );
        printf( "K " );
        Sleep( 1000 );
    } // end for
    printf( "\nDone!" );
    CoUninitialize();
    getchar();

} // main

The code use ShellExecuteEx to launch, in a loop, 64 instances of Internet Explorer with the about:blank URL. The SEE_MASK_NOCLOSEPROCESS is used to be able to then use the TerminateProcess API.

I notice two kinds of leaks:

1. Handles leaks: launching Process Explorer when the loop is finished but the program still running, I see several blocks of 64 handles (process handles, and registries handles for various keys)
2. Memory leaks: attaching the visual C++ 2017 debugger to the program, before the loop, I took a first Heap Snapshot, and a second one after the loop.I see 64 blocs of 8192 bytes, coming from windows.storage.dll!CInvokeCreateProcessVerb::_BuildEnvironmentForNewProcess()

You can read some information about the handles leaks here: ShellExecute leaks handles

Here are some screenshots: First, the PID launched and terminated:

Second: the same pids, as seen in Process Explorer:

Process Explorer also shows 64*3 open registry handles, for HKCR\.exe, HKCR\exefile and HKCR\exefile\shell\open.

One of the 64 leaked "Environment" (8192 bytes and the callstack):

Last: a screen shot of Process Explorer, showing the "Private Bytes" during the execution of the MCVE modified with a 1024 loop counter. The running time is approximately 36 minutes, the PV start at 1.1 Mo (before CoInitializeEx) and end at 19 Mo (after CoUninitialize). The value then stabilizes at 18.9

What am I doing wrong? Do I see leaks where there are none?

Answer 1

this is windows bug, in version 1803. minimal code for reproduce:

if (0 <= CoInitialize(0))
{
    SHELLEXECUTEINFO sei = {
        sizeof(sei), 0, 0, 0, L"notepad.exe", 0, 0, SW_SHOW
    };

    ShellExecuteEx( &sei );

    CoUninitialize();
}

after execute this code, can view handles for notepad.exe process and first thread - this handles of course must not exist (be closed), not closed keys

\REGISTRY\MACHINE\SOFTWARE\Classes\.exe
\REGISTRY\MACHINE\SOFTWARE\Classes\exefile

also private memory leaks exist in process after this call.

of course this bug cause permanent resource leaks in explorer.exe and any process, which use ShellExecute[Ex]

exactly research of this bug - here

The underlying issue here appears to be in windows.storage.dll. In particular, the CInvokeCreateProcessVerb object is never destroyed, because the associated reference count never reaches 0. This leaks all of the objects associated with CInvokeCreateProcessVerb, including 4 handles and some memory.

The reason the reference count never reaches 0 appears to be related to the argument change for ShellDDEExec::InitializeByShellInternal from Windows 10 1709 to 1803, executed by CInvokeCreateProcessVerb::Launch().

more concrete here we have cyclic reference of an object (CInvokeCreateProcessVerb) to itself.

more concrete error inside method CInvokeCreateProcessVerb::Launch() which call from self

HRESULT ShellDDEExec::InitializeByShellInternal(
    IAssociationElement*, 
    CreateProcessMethod,
    PCWSTR,
    STARTUPINFOEXW*, 
    IShellItem2*, 
    IUnknown*, // !!!
    PCWSTR, 
    PCWSTR, 
    PCWSTR);

with wrong 6 argument. the CInvokeCreateProcessVerb class containing internal ShellDDEExec sub-object. in windows 1709 CInvokeCreateProcessVerb::Launch() pass pointer to static_cast<IServiceProvider*>(pObj) in place 6 argument to ShellDDEExec::InitializeByShellInternal where pObj is point to instance of CBindAndInvokeStaticVerb class. but in 1803 version here passed pointer to static_cast<IServiceProvider*>(this) - so pointer to self. the InitializeByShellInternal store this pointer inside self and add reference to it. note that ShellDDEExec is sub-object of CInvokeCreateProcessVerb. so destructor of ShellDDEExec will not be called until destructor of CInvokeCreateProcessVerb not be called. but destructor of CInvokeCreateProcessVerb will be not called until it reference count reach 0. but this not happens until ShellDDEExec not release self pointer to CInvokeCreateProcessVerb which will be only inside it destructor ..

may be this more visible in pseudo code

class ShellDDEExec
{
    CComPtr<IUnknown*> _pUnk;

    HRESULT InitializeByShellInternal(..IUnknown* pUnk..)
    {
        _pUnk = pUnk;
    }
};

class CInvokeCreateProcessVerb : CExecuteCommandBase, IServiceProvider /**/
{
    IServiceProvider* _pVerb;//point to static_cast<IServiceProvider*>(CBindAndInvokeStaticVerb*)
    ShellDDEExec _exec;

    TRYRESULT CInvokeCreateProcessVerb::Launch()
    {
        // in 1709
        // _exec.InitializeByShellInternal(_pVerb); 
        // in 1803
        _exec.InitializeByShellInternal(..static_cast<IServiceProvider*>(this)..); // !! error !!
    }
};

ShellDDEExec::_pUnk hold pointer to containing object CInvokeCreateProcessVerb this pointer will be released only inside CComPtr destructor, called from ShellDDEExec destructor. called from CInvokeCreateProcessVerb destructor, called when reference count became 0, but this never happens because extra reference hold ShellDDEExec::_pUnk

so object store referenced pointer to self. after this reference count to CInvokeCreateProcessVerb never reaches 0