Byte array marshaling for PostMessage

Question

I'm trying to port some C++ code to C#, and one of the things that I need to do is use PostMessage to pass a byte array to another process' window. I'm trying to get the source code to the other program so I can see exactly what it's expecting, but in the meantime, here's what the original C++ code looks like:

unsigned long result[5] = {0};
//Put some data in the array
unsigned int res = result[0];
Text winName = "window name";
HWND hWnd = FindWindow(winName.getConstPtr(), NULL);
BOOL result = PostMessage(hWnd, WM_COMMAND, 10, res);

And here's what I have now:

[DllImport("User32.dll", SetLastError = true, EntryPoint = "FindWindow")]
public static extern IntPtr FindWindow(String lpClassName, String lpWindowName);

[DllImport("User32.dll", SetLastError = true, EntryPoint = "SendMessage")]
public static extern int SendMessage(IntPtr hWnd, int Msg, int wParam, ref COPYDATASTRUCT lParam);

[StructLayout(LayoutKind.Sequential)]
public struct COPYDATASTRUCT
{
    public int dwData;
    public int cbData;
    [MarshalAs(UnmanagedType.ByValArray, SizeConst=32)]
    public byte[] lpData;
}

public const int WM_COPYDATA = 0x4A;

public static int sendWindowsByteMessage(IntPtr hWnd, int wParam, byte[] data)
{
    int result = 0;

    if (hWnd != IntPtr.Zero)
    {
        int len = data.Length;
        COPYDATASTRUCT cds;
        cds.dwData = wParam;
        cds.lpData = data;
        cds.cbData = len;
        result = SendMessage(hWnd, WM_COPYDATA, wParam, ref cds);
    }

    return result;
}

//*****//

IntPtr hWnd = MessageHelper.FindWindow(null, windowName);
if (hWnd != IntPtr.Zero)
{
    int result = MessageHelper.sendWindowsByteMessage(hWnd, wParam, lParam);
    if (result == 0)
    {
        int errCode = Marshal.GetLastWin32Error();
    }
}

Note that I had to switch from using PostMessage in C++ to SendMessage in C#.

So what happens now is that I'm getting both result and errCode to be 0, which I believe means that the message was not processed - and indeed looking at the other application, I'm not seeing the expected response. I have verified that hWnd != IntPtr.Zero, so I think that the message is being posted to the correct window, but the message data is wrong. Any ideas what I'm doing wrong?

Update

I'm still not having any luck after trying the suggestions in the comments. Here's what I've currently got:

[DllImport("User32.dll", SetLastError = true)]
public static extern IntPtr SendMessage(IntPtr hWnd, UInt32 Msg, IntPtr wParam, IntPtr lParam);

[StructLayout(LayoutKind.Sequential)]
public struct COPYDATASTRUCT
{
    public IntPtr dwData;
    public int cbData;
    public IntPtr lpData;
}

public struct BYTEARRDATA
{
    public byte[] data;
}

public static IntPtr IntPtrAlloc<T>(T param)
{
    IntPtr retval = Marshal.AllocHGlobal(Marshal.SizeOf(param));
    Marshal.StructureToPtr(param, retval, false);
    return (retval);
}

public static void IntPtrFree(IntPtr preAllocated)
{
    //Ignores errors if preAllocated is IntPtr.Zero!
    if (IntPtr.Zero != preAllocated)
    {
        Marshal.FreeHGlobal(preAllocated); 
        preAllocated = IntPtr.Zero;
    }
}

BYTEARRDATA d;
d.data = data;
IntPtr buffer = IntPtrAlloc(d);

COPYDATASTRUCT cds;
cds.dwData = new IntPtr(wParam);
cds.lpData = buffer;
cds.cbData = Marshal.SizeOf(d);

IntPtr copyDataBuff = IntPtrAlloc(cds);
IntPtr r = SendMessage(hWnd, WM_COPYDATA, IntPtr.Zero, copyDataBuff);
if (r != IntPtr.Zero)
{
    result = r.ToInt32();
}

IntPtrFree(copyDataBuff);
copyDataBuff = IntPtr.Zero;
IntPtrFree(buffer);
buffer = IntPtr.Zero;

This is a 64 bit process trying to contact a 32 bit process, so there may be something there, but I'm not sure what.

Answer 1

The problem is that COPYDATASTRUCT is supposed to contain a pointer as the last member, and you're passing the entire array.

Take a look at the example on pinvoke.net: http://www.pinvoke.net/default.aspx/Structures/COPYDATASTRUCT.html

More info after comments:

Given these definitions:

const int WM_COPYDATA = 0x004A;

[StructLayout(LayoutKind.Sequential)]
struct COPYDATASTRUCT
{
    public IntPtr dwData;
    public int cbData;
    public IntPtr lpData;
}
[DllImport("User32.dll", SetLastError = true, EntryPoint = "FindWindow")]
public static extern IntPtr FindWindow(String lpClassName, String lpWindowName);

[DllImport("User32.dll", SetLastError = true, EntryPoint = "SendMessage")]
public static extern IntPtr SendMessage(IntPtr hWnd, int Msg, int wParam, ref COPYDATASTRUCT lParam);

I can create two .NET programs to test WM_COPYDATA. Here's the window procedure for the receiver:

protected override void WndProc(ref Message m)
{
    switch (m.Msg)
    {
        case WM_COPYDATA:
            label3.Text = "WM_COPYDATA received!";
            COPYDATASTRUCT cds = (COPYDATASTRUCT)Marshal.PtrToStructure(m.LParam, typeof(COPYDATASTRUCT)); 
            byte[] buff = new byte[cds.cbData];
            Marshal.Copy(cds.lpData, buff, 0, cds.cbData);
            string msg = Encoding.ASCII.GetString(buff, 0, cds.cbData);
            label4.Text = msg;
            m.Result = (IntPtr)1234;
            return;
    }
    base.WndProc(ref m);
}

And the code that calls it using SendMessage:

Console.WriteLine("{0} bit process.", (IntPtr.Size == 4) ? "32" : "64");
Console.Write("Press ENTER to run test.");
Console.ReadLine();
IntPtr hwnd = FindWindow(null, "JimsForm");
Console.WriteLine("hwnd = {0:X}", hwnd.ToInt64());
var cds = new COPYDATASTRUCT();
byte[] buff = Encoding.ASCII.GetBytes(TestMessage);
cds.dwData = (IntPtr)42;
cds.lpData = Marshal.AllocHGlobal(buff.Length);
Marshal.Copy(buff, 0, cds.lpData, buff.Length);
cds.cbData = buff.Length;
var ret = SendMessage(hwnd, WM_COPYDATA, 0, ref cds);
Console.WriteLine("Return value is {0}", ret);
Marshal.FreeHGlobal(cds.lpData);

This works as expected when both the sender and receiver are 32 bit processes and when they're 64 bit processes. It will not work if the two processes' "bitness" does not match.

There are several reasons why this won't work for 32/64 or 64/32. Imagine that your 64 bit program wants to send this message to a 32 bit program. The lParam value passed by the 64 bit program is going to be 8 bytes long. But the 32 bit program only sees 4 bytes of it. So that program won't know where to get the data from!

Even if that worked, the size of the COPYDATASTRUCT structure is different. In 32 bit programs, it contains two pointers and a DWORD, for a total size of 12 bytes. In 64 bit programs, COPYDATASTRUCT is 20 bytes long: two pointers at 8 bytes each, and a 4-byte length value.

You have similar problems going the other way.

I seriously doubt that you'll get WM_COPYDATA to work for 32/64 or for 64/32.

Answer 2

This will work on 32bit sender to 64bit receiver, 64bit sender to 32bit receiver. Also work from 32 to 32, and 64 to 64. You don't even need to declare COPYDATASTRUCT. Very simple:

    const int WM_COPYDATA = 0x004A;

    [DllImport("user32.dll", SetLastError = true)]
    static extern IntPtr FindWindow(String lpClassName, String lpWindowName);

    [DllImport("user32.dll", SetLastError = true)]
    static extern IntPtr SendMessage(IntPtr hWnd, int Msg, IntPtr wParam, IntPtr lParam);

    static IntPtr SendMessage(IntPtr hWnd, byte[] array, int startIndex, int length)
    {
        IntPtr ptr = Marshal.AllocHGlobal(IntPtr.Size * 3 + length);
        Marshal.WriteIntPtr(ptr, 0, IntPtr.Zero);
        Marshal.WriteIntPtr(ptr, IntPtr.Size, (IntPtr)length);
        IntPtr dataPtr = new IntPtr(ptr.ToInt64() + IntPtr.Size * 3);
        Marshal.WriteIntPtr(ptr, IntPtr.Size * 2, dataPtr);
        Marshal.Copy(array, startIndex, dataPtr, length);
        IntPtr result = SendMessage(hWnd, WM_COPYDATA, IntPtr.Zero, ptr);
        Marshal.FreeHGlobal(ptr);
        return result;
    }

    private void button1_Click(object sender, EventArgs e)
    {
        IntPtr hWnd = FindWindow(null, "Target Window Tittle");
        byte[] data = System.Text.Encoding.ASCII.GetBytes("this is the sample text");
        SendMessage(hWnd, data, 0, data.Length);
    }

    protected override void WndProc(ref Message m)
    {
        switch (m.Msg)
        {
            case WM_COPYDATA:
                byte[] b = new Byte[Marshal.ReadInt32(m.LParam, IntPtr.Size)];
                IntPtr dataPtr = Marshal.ReadIntPtr(m.LParam, IntPtr.Size * 2);
                Marshal.Copy(dataPtr, b, 0, b.Length);
                string str = System.Text.Encoding.ASCII.GetString(b);
                MessageBox.Show(str);
                // m.Result = put result here;
                return;
        }
        base.WndProc(ref m);
    }