How do I call an unmanaged function that has a char[] as OUT parameter from C#?

Question

Say, I've got that prototype of a function that is exposed on a DLL:

int CALLBACK worker (char* a_inBuf, int a_InLen,
                     char** a_pOutBuf, int* a_pOutLen,
                     char** a_pErrBuf, int* a_pErrLen)

I'm sure that it's ridiculously easy to call that DLL function from my C# code but it doesn't work with this code:

[DllImport("mydll.dll")]  
     public static extern int worker(
         [In, MarshalAs(UnmanagedType.LPArray)] byte[] inBuf,  
         int inputLen,  
         [Out, MarshalAs(UnmanagedType.LPArray)] byte[] outBuf,  
         out int outputLen,  
         [Out, MarshalAs(UnmanagedType.LPArray)] byte[] errBuf,  
         out int errorLen);

... 

int outputXmlLength = 0;
int errorXmlLength  = 0;

byte[]  outputXml = null;
byte[]  errorXml  = null;
worker(input, input.Length, output, out outputLength, error, out errorLength);

I get an access violation when I'm going to fetch the memory for output and error inside my unmanaged library (and therefore de-reference the passed pointer):

*a_ppBuffer = (char*) malloc(size*sizeof(char));

1. How do I write the DLLIMPORT statement in my C# code for this function?

2. How do I actually call the function so that a_pOutBuf and a_pErrBuf are accessible and not null from within worker (i.e. use a real double pointer)?

Answer 1

Your current definition will not work. The worker function is allocating memory inside of the function and writing to that memory.

The P/Invoke layer does not support marshaling C-style arrays that are allocated in this way, as it has no way of knowing just how large the array will be when the call returns (unlike say, a SAFEARRAY).

That's also why returning pointers to arrays from API functions is generally a bad idea, and the Windows API is written in such a way that the memory allocation is handled by the caller.

That said, you want to change the P/Invoke declaration of worker to this:

[DllImport("mydll.dll")]  
public static extern int worker(
    [In, MarshalAs(UnmanagedType.LPArray)] byte[] inBuf,  
    int inputLen,  
    ref IntPtr outBuf, 
    ref int outputLen,  
    ref IntPtr errBuf, 
    ref int errorLen
);

In doing this, you're indicating that you're going to marshal the arrays manually (the outBuf and errBuf parameters are going to be set for you); you're passing the reference to the pointer (double-indirection, that's your char**) and then have to read from them using other indicators for bounds checking (in this case, the outputLen and errorLen parameters).

You would marshal the data out of the pointers upon return like so:

int outputXmlLength = 0;
int errorXmlLength  = 0;

IntPtr output = IntPtr.Zero;
IntPtr error = IntPtr.Zero;

worker(
    input, 
    input.Length, 
    ref output, 
    ref outputLength, 
    ref error, 
    ref errorLength
);

// Get the strings.
string outputString = Marshal.PtrToStringAnsi(
    output, 
    outputLength
);
string errorString = Marshal.PtrToStringAnsi(
    error, 
    errorLength
);

That said, you have another problem. Because the memory was allocated inside the function, you have to free the memory. Since you're using malloc to allocate the memory, you need to pass the two IntPtr instances back to unmanaged code in order to have free called on them.

If you were allocating memory in unmanaged code using LocalAlloc or CoTaskMemAlloc then you could use the FreeHGlobal or FreeCoTaskMem methods respectively on the Marshal class to free the memory on the managed side.