IMFMediaBuffer from PhotoCaptureFrame

Question

I have a computer vision plugin using Hololens locatable camera on Unity 5.2. I use the class UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame with photo mode enabled at highest resolution, which is 2048x1156, in NV12 color format which is camera native format.

PhotoCaptureFrame class : https://docs.unity3d.com/2017.2/Documentation/ScriptReference/XR.WSA.WebCam.PhotoCaptureFrame.html

At the moment, I do this for my plugin to process any single photo:

PhotoCaptureFrame photo; // created correctly through Unity API

List<byte> photoBuffer = new List<byte>();
photo.CopyRawImageDataIntoBuffer(photoBuffer);

// marshal photoBuffer to plugin (and it works fine for the computer vision part)

However, it copies the raw buffer each time. I want to use the raw buffer directly from my plugin. So I tried this:

IntPtr rawBufferPtr = photo.GetUnsafePointerToBuffer(); // COM pointer to IMFMediaBuffer is incremented
// send rawBufferPtr to plugin (native function described below)
Marshal.Release(rawBufferPtr);

IMFMediaBuffer interface : https://msdn.microsoft.com/en-us/library/windows/desktop/ms696261(v=vs.85).aspx

And in my C++ plugin:

#include <Mfidl.h> // for IMFMediaBuffer support

void process_photo(void *photo_buffer_wrapper, int photo_width, int photo_height)
{
    // photo_buffer_wrapper = rawBufferPtr in managed environment

    IMFMediaBuffer *media_buffer = reinterpret_cast<IMFMediaBuffer *>(photo_buffer_wrapper);
    BYTE *photo_buffer = NULL;
    HRESULT result = media_buffer->Lock(&photo_buffer, NULL, NULL);
    if (SUCCEEDED(result))
    {
        // do image processing stuff here (with OpenCV) using photo_buffer
        media_buffer->Unlock;
    }
}

It appears fine to me. It does compile fine too. But at run time, I get an access violation and the applications crashes on Hololens.

Exception Code: 0xC0000005

Exception Information: The thread tried to read from or write to a virtual address for which it does not have the appropriate access.

Anyone sees the problem? Something to do with the way I pass the IMFMediaBuffer object from managed to unmanaged environment?

Thank you very much!

Answer 1

I will answer my own question.

The photo_buffer_wrapper is not a pointer to IMFMediaBuffer as I thought but a pointer to IUnknown. Here is the modified native function that works as intended:

// UNMANAGED ENVIRONMENT
#include <Mfidl.h> // for IMFMediaBuffer support + other COM stuff

void process_photo(void *photo_buffer_unknown, int photo_width, int photo_height)
{
    // photo_buffer_unknown = photoCaptureFrame.GetUnsafePointerToBuffer() in managed environment which is an IntPtr

    IMFMediaBuffer *media_buffer;
    if (SUCCEEDED(reinterpret_cast<IUnknown *>(photo_buffer_unknown)->QueryInterface<IMFMediaBuffer>(&media_buffer)))
    {
        BYTE* photo_buffer = NULL;
        if (SUCCEEDED(media_buffer->Lock(&photo_buffer, NULL, NULL)))
        {
            // process photo_buffer with OpenCV (wrapped in a cv::Mat)

            media_buffer->Unlock();
            media_buffer->Release(); // QueryInterface on IUnknown has incremented reference count by one
        }
    }
}

NB: The pointer returned from photoCaptureFrame.GetUnsafePointerToBuffer() still has to be released in managed environment like in my question:

// MANAGED ENVIRONMENT
IntPtr mediaBufferUnknownPtr = photoCaptureFrame.GetUnsafePointerToBuffer();
// send mediaBufferUnknownPtr to native function through extern DLL call
Marshal.Release(mediaBufferUnknownPtr)