Excel 2013 crashing

Question

I'm trying to embed Excel 2013 in a WPF app. The problem is that when I call SetWindowLongPtr in the following code, Excel 2013 crashes immediately. I digged it and found that if I comment out WS.CHILD style, it works fine, but the Excel sheet becomes readonly, which is not what I want. The same code works fine with Excel 2010.

Excel.Application _excelApp;
IntPtr _wrappedApplicationHandle;
Int64 lngStyle;
Int64 lExStyle;    

private void Button_Click_1(object sender, RoutedEventArgs e)
{
    _excelApp = new Excel.Application()
    {
        Visible = true,
        DisplayFormulaBar = true,
    };

    _wrappedApplicationHandle = new IntPtr( _excelApp.Hwnd);

    lngStyle = GetWindowLongPtr(_wrappedApplicationHandle, (int)GWL.STYLE).ToInt64();
    lngStyle &= ~(int)WS.CAPTION;
    lngStyle &= ~(int)WS.SIZEBOX;
    lngStyle |= (int)WS.MAXIMIZE;
    lngStyle |= (int)WS.CHILD; //<< crashes with this line
    lngStyle |= (int)WS.CLIPSIBLINGS;
    lngStyle |= (int)WS.CLIPCHILDREN;

    SetWindowLongPtr(new HandleRef(_excelApp, _wrappedApplicationHandle), 
                        (int)GWL.STYLE, 
                        new IntPtr(lngStyle));
    ...
}

EDIT

Some more information as I'm digging through. I tried wrapping the above code in a try/catch block to see what happens. It never reaches the catch block. Excel 2013 crashes with the infamous "Application has stopped working. Send report to MS" error. I have already turned on all Win32 / COM / C++ Exceptions in Visual Studio (through Debug menu > Exceptions dialog), but that doesn't help either. There is a Debug button in the error dialog. If I click that and open a debugger, the error msg I see is " 0xC0000005: Access violation reading location 0x0000000000000000."

I also found that commenting out the WS.CHILD line in the above code doesn't strictly make the worksheet readonly. It just blocks common keyboard/mouse input from reaching the worksheet. But some keys like the context-menu key of the keyboard still reaches there and the context menu is shown (right-clicking through the mouse doesn't work though). Similarly, I can interact with the Office Ribbon through the mouse. It appears as if only the worksheet area (the grid with the white background) is not receiving keyboard/mouse input.

EDIT 2

I just recalled that (apparently) in contrast to what Hans Passant has explained in his post below, VS2010 hosts an Excel instance in itself when you create a Excel VSTO Workbook project. Though I don't have VS2013 (Full) with me and can't confirm, I suspect VS2013 would do the same with Excel 2013 VSTO Workbook projects. Considering that VS2010 and above are all WPF applications themselves, how does that fit in the picture?

EDIT 3

That private interfaces theory suggested by @acelent (see comment below) appears to be correct. I spied into VS2010-hosted Excel instance and found that there was a new window with classname = EXCELI which is not there when we normally open Excel (normal hierarchy of windows is XLMAIN (application) > XLDESK (workspace area) > EXCEL7 (workbook)). Also that workbook is no longer available as an ActiveX object, which used to be the cases back when Office Web Components library was available (last shipped with Office 2003). So all in all, we seem to be on a dead end and I'm going to suggest @Hans's answer to my client unless someone comes up with an actual working method in the next few hours.

Answer 1

You'll need to give up on this, you cannot make it work reliably.

WS_CHILD cannot work by design, Windows has the rock-hard requirement that the parent window and its child belong to the same process. They pass messages between each other, the child will crash when it tries to dereference a pointer that belongs to the parent's process.

Windows does have some appcompat support for SetParent(), required because this was commonly done in Windows 3.x programs. The notion of processes with separate address spaces was entirely absent in that Windows version so it wasn't a problem back then. The odds that this still actually works properly after 20 years is proportional to how much the process behaves like a Windows 3.x app. It works okayish for a console window or a simple app like Notepad. Office 2013 apps are considerably beyond simple and 3.x. Having trouble with input is certainly a hallmark for this. You could tinker with AttachThreadInput() to try to solve it but you'll likely just run head-on into the next problem, including its nasty habit of causing deadlock when you debug.

Embedding Office apps used to be a strongly supported feature in Office, Microsoft intentionally designed them to be embeddable. The underlying technology was called OLE Linking and Embedding. This technology is however strongly deprecated. Support for it was intentionally left out of the .NET Framework. Office 2003 was the last version that still supported it well. Trouble started at 2007 and it was completely scrapped for 2010. It will not come back.

The way ahead here is the exact opposite one. Don't embed the Office app, let the Office app embed you. Writing add-ins for Office programs is strongly supported.