C++ dll and C# call

Question

I have a function made in C++ that calls a COM interface's function Its signature:

BOOL func(LPWSTR strIn, __out LPWSTR strOut)
{
  //initcom
  //do something
  // release pointers
}

In C#:

[DllImport("funcdll.dll")]
static extern bool func(String strIn, ref String strOut);

// use it

for(int i=0;i<10;i++)
{
   if(func(strin, strout))
   {
      //do something with strout
   }
}

I have tested my dll in a C++ console application, it works, but in C# it crashes with an unknown error.

Answer 1

You've got three problems that I can see.

1. The calling conventions don't match. Your C++ code is cdecl and your C# code is stdcall.
2. The C++ code uses wide strings, but the C# code marshals ANSI strings.
3. The second parameter doesn't match. Your C# code assumes that the C++ code returns a new pointer to a C string which the C# code then deallocates with the COM allocator. Your C++ code doesn't do this.

Now, dealing with these in more detail.

Calling conventions

This is pretty easy to fix. Simple change the C++ code to stdcall, or the C# code to cdecl. But don't do both. I'd change the C# code:

[DllImport("funcdll.dll"), CallingConvention=CallingConvention.Cdecl]

Unicode/ANSI strings

I presume you are wanting to use Unicode strings since you have explicitly selected them in the C++ code. But P/invoke defaults to marshalling ANSI strings. You can change this again in the DllImport like so:

[DllImport("funcdll.dll"), CallingConvention=CallingConvention.Cdecl, 
    CharSet=CharSet.Unicode]

Returning a string from C++ to C#

Your current C++ function declaration is so:

BOOL func(LPWSTR strIn, __out LPWSTR strOut)

The __out decorator has no real effect, other than documenting that you want to modify the buffer pointed to by strOut and have those modifications returned to the caller.

Your C# declaration is:

static extern bool func(String strIn, ref String strOut);

Now, ref String strOut simply does not match. A ref string parameter matches this in C++:

BOOL func(LPWSTR strIn, LPWSTR *strOut)

In other words the C# code is expecting you to return a new pointer. In fact it will then proceed to deallocate the buffer you returned in strOut by calling CoTaskMemFree. I'm confident that's not what you want.

Your original C++ code can only return a string to the C# code by modifying the buffer that was passed to it. That code would look like this:

BOOL func(LPWSTR strIn, __out LPWSTR strOut)
{
    ...
    wcscpy(strOut, L"the returned string");
    ...
}

If this is what you want then you should allocate a sufficient buffer in C# in a StringBuilder object.

[DllImport("funcdll.dll"), CallingConvention=CallingConvention.Cdecl, 
    CharSet=CharSet.Unicode]
static extern bool func(string strIn, StringBuilder strOut);
...
StringBuilder strOutBuffer = new StringBuilder(128);
bool res = func("input string", strOutBuffer);
string strOut = StringBuilder.ToString();

If you simply cannot decide in the C# code how big a buffer you need then your best bet is to use a BSTR to marshal strOut. See this answer for details.

Answer 2

It's hard for me to tell without seeing the details of your C++ method... but, I've never had much luck using String with P/Invoke.

Try using IntPtr instead of String, and use Marshal.PtrToStringUni for the outgoing string, and marshal your managed string into unmanaged land with Marshal.StringToHGlobalUni and, after the function call, make sure to free the unmanaged string with Marshal.FreeHGlobal.

Also, coming from C99 land, my bools don't work with .NET bools... I don't know why. I have to use byte and check == 1. Don't know if you'll run into this with C++... but, if you want my advice on a place to start which seems least-breakable in my experience, here ya go:

[DllImport("funcdll.dll")]
static extern byte func(IntPtr strIn, out IntPtr strOut);

// use it
string myString = "Testing";

IntPtr stringOut;
IntPtr stringIn = Marshal.StringToHGlobalUni(myString);

   if(func(stringIn, out stringOut) == 1)
   {
      //do something with strout
      string stringOutValue = Marshal.PtrToStringUni(stringOut);
      // Depending on how you dealt with stringOut in your
      // unmanaged code, you may have to: Marshal.FreeCoTaskMem(stringOut);
      // or Marshal.FreeHGlobal(stringOut) if you're returning
      // an owning reference to a copied string.
   }

Marshal.FreeHGlobal(stringIn);