How do I call this c function in c# (unmarshalling return struct)?

Question

I want to use c# interop to call a function from a dll written in c. I have the header files. Take a look at this:

enum CTMBeginTransactionError {
    CTM_BEGIN_TRX_SUCCESS = 0,
    CTM_BEGIN_TRX_ERROR_ALREADY_IN_PROGRESS,
    CTM_BEGIN_TRX_ERROR_NOT_CONNECTED
};

#pragma pack(push)
#pragma pack(1)
struct CTMBeginTransactionResult {
    char *                        szTransactionID;
    enum CTMBeginTransactionError error;
};

struct CTMBeginTransactionResult ctm_begin_customer_transaction(const char * szTransactionID);

How do I call ctm_begin_customer_transaction from c#. The const char * mapps well to string, but despite various attempts (looking at stackoverflow and other sites), I fail to marshal the return structure. If I define the function to return IntPtr it works ok...

Edit I changed the return type to IntPtr and use: CTMBeginTransactionResult structure = (CTMBeginTransactionResult)Marshal.PtrToStructure(ptr, typeof(CTMBeginTransactionResult)); but it throws AccessViolationException

I also tried:

IntPtr ptr = Transactions.ctm_begin_customer_transaction("");
int size = 50;
byte[] byteArray = new byte[size];
Marshal.Copy(ptr, byteArray, 0, size);
string stringData = Encoding.ASCII.GetString(byteArray);

stringData == "70e3589b-2de0-4d1e-978d-55e22225be95\0\"\0\0\a\0\0\b\b?" at this point. The "70e3589b-2de0-4d1e-978d-55e22225be95" is the szTransactionID from the struct. Where is the Enum? Is it the next byte?

Answer 1

There's a memory management problem hidden in this struct. Who owns the C string pointer? The pinvoke marshaller will always assume that the caller owns it so it will try to release the string. And passes the pointer to CoTaskMemFree(), same function as the one called by Marshal.FreeCoTaskMem(). These functions use the COM memory allocator, the universal interop memory manager in Windows.

This rarely comes to a good end, C code does not typically use that allocator unless the programmer designed his code with interop in mind. In which case he'd never have used a struct as a return value, interop always works much less trouble-free when the caller supplies buffers.

So you cannot afford to let the marshaller do its normal duty. You must declare the return value type as IntPtr so it doesn't try to release the string. And you must marshal it yourself with Marshal.PtrToStructure().

That however still leaves the question unanswered, who owns the string? There is nothing you can do to release the string buffer, you don't have access to the allocator used in the C code. The only hope you have is that the string wasn't actually allocated on the heap. That's possible, the C program might be using string literals. You need to verify that guess. Call the function a billion times in a test program. If that doesn't explode the program then you're good. If not then only C++/CLI can solve your problem. Given the nature of the string, a "transaction ID" ought to change a lot, I'd say you do have a problem.

Answer 2

I hate to answer my own question, but I found the solution to marshal the resulting struct. The struct is 8 bytes long (4 bytes for the char * and 4 bytes for enum). Marshalling the string does not work automatically, but the following works:

// Native (unmanaged)
public enum CTMBeginTransactionError
{
    CTM_BEGIN_TRX_SUCCESS = 0,
    CTM_BEGIN_TRX_ERROR_ALREADY_IN_PROGRESS,
    CTM_BEGIN_TRX_ERROR_NOT_CONNECTED
};

// Native (unmanaged)
[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
internal struct CTMBeginTransactionResult
{
    public IntPtr szTransactionID;
    public CTMBeginTransactionError error;
};

// Managed wrapper around native struct
public class BeginTransactionResult
{
    public string TransactionID;
    public CTMBeginTransactionError Error;

    internal BeginTransactionResult(CTMBeginTransactionResult nativeStruct)
    {
        // Manually marshal the string
        if (nativeStruct.szTransactionID == IntPtr.Zero) this.TransactionID = "";
        else this.TransactionID = Marshal.PtrToStringAnsi(nativeStruct.szTransactionID);

        this.Error = nativeStruct.error;
    }
}

[DllImport("libctmclient-0.dll")]
internal static extern CTMBeginTransactionResult ctm_begin_customer_transaction(string ptr);

public static BeginTransactionResult BeginCustomerTransaction(string transactionId)
{
    CTMBeginTransactionResult nativeResult = Transactions.ctm_begin_customer_transaction(transactionId);
    return new BeginTransactionResult(nativeResult);
}

The code works, but I still need to investigate, if calling the unmanaged code results in memory leaks.