Windows Media Foundation using IMFTransform to decode mp4 movie frames to 2D textures

Question

I'm trying to decode an mp4 video using Windows Media Foundation classes and converting frames in to 2D textures that can be used by a DirectX shader for rendering. I've been able to read the source stream using MFCreateSourceReaderfromURL and been able to read the media type of the stream which has its major type MFMEdiaType_Video and minor type as MFVideoFormat_H264 as expected.

I'm now needing to convert this format in to an RGB format that could be used to initialise a D3D11_TEXTURE2D resource and resource view which can then be passed to a HLSL pixel shader for sampling. I've tired using the IMFTransform class to do the conversion for me but when I try to set the output type on the transform to any MFVideoFormat_RGB variant I get an error. I've also tried setting a new output type on the source reader and just Sampling that hoping to get a sample in the correct format but again I've had no luck.

So my questions would be:

• Is this type of conversion possible?

• Can this be done through the IMFTransform/SourceReader classes like I've tired above and do I just need to tweak the code or do I need to do this type of conversion manually?

• Is this the best way to go about feeding video texture data in to a shader for sampling or is there an easier alternative that I've not thought about.

The OS being used is Windows 7 so I can't use the SourceReaderEx or ID3D11VideoDevice interface because as far as I'm aware these solutions only seem available on Windows 8.

Any help/pointers in the right direction would be greatly appreciated, I can also provide some source code if necessary.

Answer 1

Is this type of conversion possible?

Yes it is possible. Stock H.264 Video Decoder MFT is "Direct3D aware" which means it can decode video into Direct3D 9 surfaces/Direct3D 11 textures leveraging DXVA. Or, if hardware capabilities are insufficient there is a software fallback mode too. You are interested in getting the output delivered right into texture for performance reasons (otherwise you would have to load this data yourself spending CPU and video resources on that).

Can this be done through the IMFTransform/SourceReader classes like I've tired above and do I just need to tweak the code or do I need to do this type of conversion manually?

IMFTransform is abstract interface. It is implemented by H.264 decoder (as well as other MFTs) and you can use it directly, or you can use higher level Source Reader API to get it manage video reading from file and decoding using this MFT.

That is, MFT and Source Reader are not actually exclusive alternate option but instead a higher and lower level APIs. MFT interface is offered by decoder and you are responsible to feed H.264 in and drain the decoded output. Source Reader manages the same MFT and adds file reading capability.

Source Reader itself is available in Windows 7, BTW (even on Vista, might be limited in feature set compared to newer OSes though).