COM cannot start out-of-process .Net server compiled as AnyCPU

Question

I am trying to get COM to start my out-of-process .NET COM server. It works if the server process is compiled with x64, but if I use AnyCPU (which is what I want) then it hangs for a while and eventually fails with 0x80080005 (CO_E_SERVER_EXEC_FAILURE). How can I get this to work?

• I am running on a 64-bit machine: Windows 7 with Visual Studio 2008 SP1.
• I can see in Task Manager that it does start my server. So I guess the problem is in the communications between COM and the server (class registration).
• My test client application is written in C#, but it doesn't matter whether it is compiled for x86 or x64. The problem also occurs with something written in 32-bit C++.
• If I rebuild the server using x64 and run it, and then rebuild back as AnyCPU, then COM can start it. A reboot will take me back to the original situation. Perhaps COM doesn't know in advance what bitness is going to be used, and a previous execution helps.
• I found Andy McMullen's blog post and tried passing CLSCTX_ACTIVATE_64_BIT_SERVER to CoCreateInstance(), but that triggers a failure earlier: 0x80040154 (REGDB_E_CLASSNOTREG). Am I doing something wrong in my COM registration? You can see below that it is very simple. Registration occurs when running in 64 bits, and the problem occurs when the client is 64 bits, so Wow6432Node should not be involved.

Another chap has had a similar problem, but the MSFT answer is confusing. He seems to be suggesting it can only work via DCOM (see link) or COM+. I suspect either will be an awful lot of work, and substantially worse than distributing my .exe built as x64 and x86.

You may be wondering why I am implementing IPersistFile. It is because my real problem is to get BindMoniker() working from a 32-bit C++ program to my AnyCPU .Net program. I have reduced my problem to the simpler example presented here.

Here is the client code:

public partial class Form1 : Form
{
    public Form1()
    {
        InitializeComponent();
    }

    [DllImport("ole32.dll", ExactSpelling = true, PreserveSig = false)]
    [return: MarshalAs(UnmanagedType.Interface)]
    static extern object CoCreateInstance(
       [In, MarshalAs(UnmanagedType.LPStruct)] Guid rclsid,
       [MarshalAs(UnmanagedType.IUnknown)] object pUnkOuter,
       CLSCTX dwClsContext,
       [In, MarshalAs(UnmanagedType.LPStruct)] Guid riid);

    [Flags]
    enum CLSCTX : uint
    {
        CLSCTX_LOCAL_SERVER = 0x4,
        CLSCTX_ACTIVATE_64_BIT_SERVER = 0x80000,
    }

    private void Form1_Load(object sender, EventArgs e)
    {
        IPersistFile pf = (IPersistFile)CoCreateInstance(
            new Guid("1984D314-FC8D-44bc-9146-8A13500666A6"),
            null,
            CLSCTX.CLSCTX_LOCAL_SERVER,
            new Guid("0000010b-0000-0000-C000-000000000046"));  // IPersistFile
        pf.Load("c:\\bozo", 0);
    }
}

and here is the server:

static class Program
{
    [STAThread]
    static void Main()
    {
        if (Environment.CommandLine.Contains("/reg")) {
            RegistryKey cls = Registry.LocalMachine.CreateSubKey(String.Format(
                "SOFTWARE\\Classes\\CLSID\\{0}", PersistFile.ClassID.ToString("B")));
            cls.SetValue("InprocHandler32", "Ole32.dll");
            RegistryKey ls32 = cls.CreateSubKey("LocalServer32");
            ls32.SetValue(null, '"' + Application.ExecutablePath + '"');
            ls32.SetValue("ServerExecutable", Application.ExecutablePath);
        }

        Application.EnableVisualStyles();
        Application.SetCompatibleTextRenderingDefault(false);

        RegistrationServices reg = new RegistrationServices();
        reg.RegisterTypeForComClients(
            typeof(PersistFile),
            RegistrationClassContext.LocalServer,
            RegistrationConnectionType.MultipleUse);

        Application.Run(new Form1());
    }
}

[ComVisible(true),
 Guid("1984D314-FC8D-44bc-9146-8A13500666A6"),
 ClassInterface(ClassInterfaceType.None)]
public class PersistFile : IPersistFile
{
    public static Guid ClassID
    {
        get
        {
            GuidAttribute a = (GuidAttribute)typeof(PersistFile).GetCustomAttributes(typeof(GuidAttribute), false)[0];
            return new Guid(a.Value);
        }
    }

    #region IPersistFile
    public void GetClassID(out Guid pClassID)
    {
        MessageBox.Show("GetClassID");
        pClassID = ClassID;
    }

    public int IsDirty()
    {
        MessageBox.Show("IsDirty");
        return 1;
    }

    public void Load(string pszFileName, int dwMode)
    {
        MessageBox.Show(String.Format("Load {0}", pszFileName));
    }

    public void Save(string pszFileName, bool fRemember)
    {
        MessageBox.Show("Save");
        throw new NotImplementedException();
    }

    public void SaveCompleted(string pszFileName)
    {
        MessageBox.Show("SaveCompleted");
        throw new NotImplementedException();
    }

    public void GetCurFile(out string ppszFileName)
    {
        MessageBox.Show("GetCurFile");
        throw new NotImplementedException();
    }
    #endregion
}

Answer 1

I guess the problem is at run-time. I have created a COM Server which registers using a C++ library (registration is performed flawlessly). I have run into problems when switching to AnyCPU from .NET (CS).

The architecture:

• C++ library interfacing COM (built on both x64 and x86 platforms)
• .NET library wrapper (CS) (correctly instantiates the required x64/x86 C++ library)
• .NET application (CS) - COM client or COM server

Ugly things happen when registering the .NET application built as "AnyCPU". Once the COM Client invokes the COM Server through DCOM, the server application starts but the client receives error that the COM Server could not be started.

I went some steps further, analyzed registration data with procmon and other tools and I reached the same conclusion:

• x86 registers the classes in CLASSES\Wow6432Node
• x64 and AnyCPU register the classes in CLASSES (on a x64 windows machine, exactly the same keys; I bet that x86 and AnyCPU would register the same on an x86 machine)

Now, I did some more experiments: the x86/x64/AnyCPU COM Client can connect with no problems to any x86/x64 COM Server but it cannot connect anyhow to an AnyCPU COM Server...

I then performed the following test cases:

1. Have the x86 COM Server register, replace the executable with the AnyCPU COM Server: COM Client was starting the x86 COM Server, but no communication... it was starting the server over and over again..
2. Have the x64 COM Server register, replace the executable with the AnyCPU COM Server: COM Client was starting the x64 COM Server, but no communication... it was starting the server over and over again..
3. Have the AnyCPU COM Server register, replace the executable with the x86 COM Server: COM Client was able to successfully start the and connect to the x86 COM Server.
4. Have the AnyCPU COM Server register, replace the executable with the x64 COM Server: COM Client was able to successfully start the and connect to the x64 COM Server.
5. Have the x86 COM Server register, replace the executable with the x64 COM Server: COM Client was able to successfully start the and connect to the x64 COM Server.
6. Have the x64 COM Server register, replace the executable with the x86 COM Server: COM Client was able to successfully start the and connect to the x86 COM Server.

Where the hell is the communication problem? This is very odd... None of the presented solutions (CLSCTX_ACTIVATE_64_BIT_SERVER, PreferredServerBitness or corflags) helped.

Anyone else did some progress on this matter? Should we contact Microsoft?

Answer 2

Try to use the RegistrationServices class to register your com assembly. It will also choose the correct registry pathes and do some other things.

Example:

Assembly currentAssembly = Assembly.GetExecutingAssembly();
System.Runtime.InteropServices.RegistrationServices regAsm = new System.Runtime.InteropServices.RegistrationServices();
bool isRegistered = regAsm.RegisterAssembly(currentAssembly, System.Runtime.InteropServices.AssemblyRegistrationFlags.SetCodeBase);

Also I think, that .NET client assemblies have some trouble according to .NET com servers, but I can't find any resource for it...

Hope, it will help...