RtlCompressBuffer API in C#

Question

I'm trying to use the RtlGetCompressionWorkSpaceSize and RtlCompressBuffer functions in a C# project.

Here is what I have so far:

class Program
{
    const uint COMPRESSION_FORMAT_LZNT1 = 2;
    const uint COMPRESSION_ENGINE_MAXIMUM = 0x100;

    [DllImport("ntdll.dll")]
    static extern uint RtlGetCompressionWorkSpaceSize(uint CompressionFormat, out uint pNeededBufferSize, out uint Unknown);

    [DllImport("ntdll.dll")]
    static extern uint RtlCompressBuffer(uint CompressionFormat, byte[] SourceBuffer, uint SourceBufferLength, out byte[] DestinationBuffer,
        uint DestinationBufferLength, uint Unknown, out uint pDestinationSize, IntPtr WorkspaceBuffer);

    static void Main(string[] args)
    {
        uint dwSize = 0;
        uint dwRet = 0;
        uint ret = RtlGetCompressionWorkSpaceSize(COMPRESSION_FORMAT_LZNT1 | COMPRESSION_ENGINE_MAXIMUM, out dwSize, out dwRet);

        IntPtr pMem = Marshal.AllocHGlobal((int)dwSize);
        byte[] buffer = new byte[1024];
        byte[] outBuf = new byte[1024];
        uint destSize = 0;
        ret = RtlCompressBuffer(COMPRESSION_FORMAT_LZNT1 | COMPRESSION_ENGINE_MAXIMUM, buffer, 1024, out outBuf, 1024, 0, out destSize, pMem);

        Console.Write(ret.ToString());
        Console.Read();
    }
}

RtlGetCompressionWorkSpaceSize works since it returns 0 (NT success code) but when I call RtlCompressBuffer I get a Memory Access Violation error.

EDIT: With help from David's answer I've fixed the issue and the correct code is below.

    const ushort COMPRESSION_FORMAT_LZNT1 = 2;
    const ushort COMPRESSION_ENGINE_MAXIMUM = 0x100;

    [DllImport("ntdll.dll")]
    static extern uint RtlGetCompressionWorkSpaceSize(ushort CompressionFormat, out uint pNeededBufferSize, out uint Unknown);

    [DllImport("ntdll.dll")]
    static extern uint RtlCompressBuffer(ushort CompressionFormat, byte[] SourceBuffer, int SourceBufferLength, byte[] DestinationBuffer,
        int DestinationBufferLength, uint Unknown, out int pDestinationSize, IntPtr WorkspaceBuffer);

    [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    internal static extern IntPtr LocalAlloc(int uFlags, IntPtr sizetdwBytes);

    [DllImport("kernel32.dll", SetLastError = true)]
    static extern IntPtr LocalFree(IntPtr hMem);

    internal static byte[] Compress(byte[] buffer)
    {
        var outBuf = new byte[buffer.Length * 6];
        uint dwSize = 0, dwRet = 0;
        uint ret = RtlGetCompressionWorkSpaceSize(COMPRESSION_FORMAT_LZNT1 | COMPRESSION_ENGINE_MAXIMUM, out dwSize, out dwRet);
        if (ret != 0)
        {
            return null;
        }

        int dstSize = 0;
        IntPtr hWork = LocalAlloc(0, new IntPtr(dwSize));
        ret = RtlCompressBuffer(COMPRESSION_FORMAT_LZNT1 | COMPRESSION_ENGINE_MAXIMUM, buffer,
            buffer.Length, outBuf, outBuf.Length, 0, out dstSize, hWork);
        if (ret != 0)
        {
            return null;
        }

        LocalFree(hWork);

        Array.Resize(ref outBuf, dstSize);
        return outBuf;
    }

Answer 1

You are very nearly there. The problem is this part of your P/invoke for RtlCompressBuffer:

out byte[] DestinationBuffer

The default marshalling for byte[] is for the array contents to marshalled in both directions, from managed to unmanaged, and then back again when the function returns. The C definition of RtlCompressBuffer is annotated with __out but that means that the array contents are __out rather than the pointer being __out.

Change your P/invoke to

byte[] DestinationBuffer

and similarly in the call to RtlCompressBuffer change out outBuf to outBuf and you should be good to go.

Be warned that your code as it stands will return an status code of STATUS_BUFFER_ALL_ZEROS so don't be tricked into thinking that this non-zero return value indicates failure.

One final point, the first parameter to both P/invokes, CompressionFormat, should be declared as ushort.