What is the most performant way to render unmanaged video frames in WPF?

Question

I'm using FFmpeg library to receive and decode H.264/MPEG-TS over UDP with minimal latency (something MediaElement can't handle).

On a dedicated FFmpeg thread, I'm pulling PixelFormats.Bgr32 video frames for display. I've already tried InteropBitmap:

_section = CreateFileMapping(INVALID_HANDLE_VALUE, IntPtr.Zero, PAGE_READWRITE, 0, size, null);
_buffer = MapViewOfFile(_section, FILE_MAP_ALL_ACCESS, 0, 0, size);
Dispatcher.Invoke((Action)delegate()
{
    _interopBitmap = (InteropBitmap)Imaging.CreateBitmapSourceFromMemorySection(_section, width, height, PixelFormats.Bgr32, (int)size / height, 0);
    this.Source = _interopBitmap;
});

And then per frame update:

Dispatcher.Invoke((Action)delegate()
{
    _interopBitmap.Invalidate();
});

But performance is quite bad (skipping frames, high CPU usage etc).

I've also tried WriteableBitmap: FFmpeg is placing frames in _writeableBitmap.BackBuffer and per frame update:

Dispatcher.Invoke((Action)delegate()
{
    _writeableBitmap.Lock();
});
try
{
    ret = FFmpegInvoke.sws_scale(...);
}
finally
{
    Dispatcher.Invoke((Action)delegate()
    {
        _writeableBitmap.AddDirtyRect(_rect);
        _writeableBitmap.Unlock();
    });
}

Experiencing almost the same performance issues (tested with various DispatcherPriority).

Any help will be greatly appreciated.

Answer 1

I know it is too late, but I write this answer for those folks who are struggling to solve this problem. Recently, I have done a rendering project using InteropBitmap in which I was able to run about 16 media player components in a WPF window at the same time on a core i7 1.6 Ghz CPU +8Gb RAM Laptop, with 25fps. Here are the tips I took for performance tweaking:

First of all, I did not let GC handle my video packets. I Allocated memory using Marshal.AllocateHGlobal wherever I needed to instantiate a video frame and Disposed using Marshal.FreeHGlobal as soon as I did the rendering.

Secondly, I created a dispatcher thread for each individual media player. For more information, read "https://blogs.msdn.microsoft.com/dwayneneed/2007/04/26/multithreaded-ui-hostvisual/".

Thirdly, for aspect ratio, and generally the resizing purposes, I used native EmguCV library. This library helped me a lot on performance rather than using bitmaps and overlays and etc.

I think these steps help everyone that needs to render manually using InteropBitmap or etc.