H.264 Frames Memory Leak With Some Decoders

Question

I'm receiving an H.264 stream from a DVR using its SDK. There were memory leaks and i thought it was the SDK causing all the leaks. But when i recorded the stream and played the frames one by one reading from the disk (without any 3rd party dlls involved), i noticed that the problem is not the dll but the stream itself.

Strange enough, DivX H264 Decoder is the only codec which doesn't cause a memory leak but when the stream runs for a long time, sometimes DivX decoder crashes as well. I'd prefer to use Microsoft DTV-DVD Video Decoder but it causes huge memory leaks and drops a lot of frames. Many other H.264 decoders I've tried behaves the same way.

I examined the h.264 frames using some h.264 parsers comparing with some other problem-free streams but i didn't notice anything obvious from the logs.

Since my problem is about the h.264 frames structure, i've prepared a source filter named FramesFromFileSourceFilter which you can download below.

http://www.akaydin.com/directshow/FramesFromFileSourceFilter.zip

It's a Visual Studio 2008 project and all dependencies are included in the zip file in relatively located folders (including the h.264 frames). So, all you need to do is to compile the project, register the output with regsvr32.exe and run the filter with any h.264 decoder you want from GraphEdit or GraphStudio. Example graphs are below.

Also h264 frames are available as a single raw h264 file at the link below which is playable by VLC (with wrong FPS since original was 12 FPS).

http://www.akaydin.com/directshow/stream.zip

Question:

What might be causing the memory leak problems with many famous H264 Decoders except DivX decoder. What is wrong with this stream?

Update 1

Reading data thread is removed and functionality moved into FillBuffer without using any buffers and flags. Problem remains the same.

http://www.akaydin.com/directshow/FramesFromFileSourceFilterUpdate1.zip

Update 2

Update1 was using Sleep() in FillBuffer() function which was causing some problems. Now i removed the Sleep() and used SetTime() to have ~12 FPS. This also solved Microsoft DTV-DVD Video Decoder's dropping frame issues but didn't solve memory problems.

http://www.akaydin.com/directshow/FramesFromFileSourceFilterUpdate2.zip

Memory increase occurs at Working Set only. Virtual Bytes and Private Bytes seem to be stable. What might be causing continuous Working Set memory increment which only happens with Microsoft DTV-DVD Video Decoder?

Answer 1

You don't do any synchronization around your variables

BYTE* m_buffer;
DWORD m_bufferSize;
bool isFrameReady;

And they are used from two concurrent threads. You just leak your memory by this inaccurate allocation/deallocation AND/OR letting your code crash on access violation. Debug build of your DLL indicates this by showing you "heap corruption" alert as you run your test. The runtime behavoir might vary with decoders and environment, however this is definitely a severe bug to be fixed.

For instance, you can use CAutoLock cAutoLock(m_pLock); in your thread that fills buffer to prevent streaming thread access while you are reading data from file.

Note that you read next frame into the same buffer pointer without a check whether previously allocated memory is freed or not, you just overwrite the pointer possibly leaving a leak.

Memory Leak/Working Set Update: Now, when code issues are sorted out, the unwanted runtime behavior is an increase in Working Set size. This is not a leak. It is an indication that Windows considers the process as a sort of priority (why not? it is active and works with memory) and throws more real pages towards this process to facilitate its performance. See this answer on good explanation of how process memory metrics correspond to memory leaks in an application.

The difference between decoders you might be seeing is likely to be caused by the fact that some decoders are good with smaller amount of buffers, or reuse them more actively, e.g. prefer to take the same buffer out of a pool rather than picking one by one through all available.