MFC application crashing in ProcessShellCommand() when file to open specified on command line

Question

The problem I need to solve is how to use the MFC function ProcessShellCommand() in the InitInstance() of a CWinApp to process a File Open with a specific path when the application to open the file is being launched by another application.

I have an MFC MDI (Multiple Document Interface) application that is started by another application with a command line using ShellExecute() containing the path to a file to open. When compiled with Visual Studio 2005, I do not see a problem with the launched application. When compiled with Visual Studio 2013, the application that is started crashes and I never see the application window.

Running in the debugger, I see an error dialog that has a title of "Microsoft Visual C++ Runtime Library" with an error message of "Debug Assertion Failed!" specifing the mfc120ud.dll and file of src\mfc\filelist.cpp line: 221

At this point I can attach to the application process then click the Retry button of the dialog. Then when I continue I see a Visual Studio error dialog from an unhandled exception that seems to be generated by KernelBase.dll.

Unhandled exception at 0x76EBC54F in NHPOSLM.exe: Microsoft C++ exception: CInvalidArgException at memory location 0x0014F094.

If I click the Continue button, I get another "Debug Assertion Failed" this time from src\mfc\filelist.cpp line: 234

After making a source change to perform a Sleep() in order to use the Debug->Attach to process Visual Studio 2013 command I was able to use the debugger to look at various data areas and step through code.

At one point, after stepping over the ProcessShellCommand() function and seeing the exception, when the thread returned to the statement after the function call, I used the set source line debugger command to set the current line back to the function call and stepped over it again. This time there was no exception and when I allowed the thread to continue, the application came up with the correct file open.

Then I found this article, ProcessShellCommand and the View and Frame Windows which states the following:

The problem is that the code in ProcessShellCommand() opens the document file before it finishes creating the frame and view windows. Those windows exist but there is no way to access them because the frame window pointer is not saved to an app-wide variable until after the document is open.

The solution provided in the article is to call ProcesShellCommand() twice as in the following code segment.

CCommandLineInfo cmdInfo;

if( !ProcessShellCommand( cmdInfo ) )
    return FALSE;

ParseCommandLine( cmdInfo );

if( cmdInfo.m_nShellCommand != CCommandLineInfo::FileNew )
{
    if (!ProcessShellCommand( cmdInfo ) )
        return FALSE;
}

I have tried this approach in my application and it does indeed open the document and seems to process everything correctly. The problem is that while this works for an SDI (Single Document Interface) type of MFC application for an MDI (Multiple Document Interface) type of MFC application you will see two document windows, an empty one as created by File New and the one actually wanted created by File Open.

I have also found that using the debugger to attach to the application process and then stepping through slowly, if I let the launched application Continue after the exception dialogs, the application will finish coming up with the requested file. However if not in the debugger, the launched application's main window will not display.

So there appears to be some kind of a race condition for the environment to be ready for the launched application to have its run time environment fully initialized.

For an explanation of the ProcessShellCommand() function see CWinApp::ProcessShellCommand which describes the basic process for command line processing as:

1. After being created in InitInstance, the CCommandLineInfo object is passed to ParseCommandLine.
2. ParseCommandLine then calls CCommandLineInfo::ParseParam repeatedly, once for each parameter.
3. ParseParam fills the CCommandLineInfo object, which is then passed to ProcessShellCommand.
4. ProcessShellCommand handles the command-line arguments and flags.

The specific source we are using in the InitInstance() is:

// Register the application's document templates.  Document templates
//  serve as the connection between documents, frame windows and views.

CMultiDocTemplate* pDocTemplate;
pDocTemplate = new CMultiDocTemplate(
    IDR_NEWLAYTYPE,
    RUNTIME_CLASS(CNewLayoutDoc),
    RUNTIME_CLASS(CChildFrame), // custom MDI child frame
    RUNTIME_CLASS(CNewLayoutView/*CLeftView*/));
AddDocTemplate(pDocTemplate);

// create main MDI Frame window
CMainFrame* pMainFrame = new CMainFrame;
if (!pMainFrame->LoadFrame(IDR_MAINFRAME))
    return FALSE;
m_pMainWnd = pMainFrame;

// Parse command line for standard shell commands, DDE, file open
CLOMCommandLineInfo cmdInfo;
/*initialize language identifier to English so we wont have garbage if no language 
flag is set on teh command line*/
cmdInfo.lang = LANG_ENGLISH;
cmdInfo.sublang = SUBLANG_ENGLISH_US;
//CCommandLineInfo cmdInfo;
ParseCommandLine(cmdInfo);

BOOL success = pMainFrame->ProcessCmdLineLang(cmdInfo.lang, cmdInfo.sublang);
if(!success){
    AfxMessageBox(IDS_CMDLINE_LANG_NF,MB_OK,0);
}
// Dispatch commands specified on the command line
if (!ProcessShellCommand(cmdInfo))
    return FALSE;

// The main window has been initialized, so show and update it.
pMainFrame->ShowWindow(SW_SHOWNORMAL);
pMainFrame->UpdateWindow();

I do not like the solution provided in the article of calling ProcessShellCommand() twice as it appears untidy. It does not provide what I need for an MDI application. I do not know why this code seems to work fine with VS 2005 and cause an error in VS2013.

Finally I came across this posting in codeproject, Debug Assertion Error Visual Studio 2010, which indicated that a similar assertion error involving src\mfc\filelist.cpp was traced to adding a file path to the Recent File List when the file path contained an asterisk.

When I use the debugger to look at the cmdInfo object there is a member, (*((CCommandLineInfo*)(&(cmdInfo)))).m_strFileName, which contains a value of L"C:\Users\rchamber\Documents\ailan_221.dat". This is the correct path from the command line provided by the application that is starting the launched application with ShellExecute().

Note: each of the back slashes in the string are actually double backslashes in the debug watch. So to render in stack overflow properly I need to add additional backslashes as in L"C:\\Users\\rchamber\\Documents\\ailan_221.dat" however the double backslash seems to be what the debugger uses to represent a single backslash character.

Edit 3/23/2016 - note concerning source history

One additional bit of information is the source history for this application. The original application was created using Visual Studio 6.0 then moved to Visual Studio 2005. The InitInstance() method of the CWinApp has not been modified to any degree since it was originally created.

Answer 1

After using Visual Studio 2013 to generate a new MFC MDI (Multiple Document Interface) application to compare between the application with which I am having problems launching and the new, generated source code, I have a solution.

The main difference between launching properly and not launching properly seems to be a requirement for initializing COM. The following specific source code was put into the InitInstance() of the application being launched and the application is now working successfully. Part of the source code changes is a call to initialize COM.

// InitCommonControlsEx() is required on Windows XP if an application
// manifest specifies use of ComCtl32.dll version 6 or later to enable
// visual styles.  Otherwise, any window creation will fail.
INITCOMMONCONTROLSEX InitCtrls;
InitCtrls.dwSize = sizeof(InitCtrls);
// Set this to include all the common control classes you want to use
// in your application.
InitCtrls.dwICC = ICC_WIN95_CLASSES;
InitCommonControlsEx(&InitCtrls);

CWinApp::InitInstance();

// Initialize OLE libraries
if (!AfxOleInit())
{
    AfxMessageBox(IDP_OLE_INIT_FAILED);
    return FALSE;
}

AfxEnableControlContainer();

// AfxInitRichEdit2() is required to use RichEdit control   
// AfxInitRichEdit2();

While the Visual Studio 2005 compiled application did not demonstrate this problem, I do want to keep the source as compiled by Visual Studio 2005 and Visual Studio 2013 as similar as possible. I made the same source change in the Visual Studio 2005 source tree and it works properly in the Visual Studio 2005 source tree as well.

Using Visual Studio 2005 and creating an empty MFC application for MDI generates source code similar to the above.