Media Foundation - How to change frame-size in MFT (Media Foundation Transform)

Question

I am trying to implement an MFT which is able to rotate a video. The rotation itself would be done inside a transform function. For that i need to change the output frame size but i don´t know how to do that.

As a starting point, i used the MFT_Grayscale example given by Microsoft. I included this MFT in a partial topology as a transform node

HRESULT Player::AddBranchToPartialTopology(
    IMFTopology *pTopology,
    IMFPresentationDescriptor *pSourcePD,
    DWORD iStream
    )
{
    ...
    IMFTopologyNode pTransformNode = NULL;
    ...
    hr = CreateTransformNode(CLSID_GrayscaleMFT, &pTransformNode);
    ...
    hr = pSourceNode->ConnectOutput(0, pTransformNode, 0);
    hr = pTransformNode->ConnectOutput(0, pOutputNode, 0);
    ...
}

This code is working so far. The grayscale mft is applied and working as expected. Anyway i want to change this mft to handle video rotation. So lets assume i want to rotate a video by 90 degrees. For that the width and height of my input frame have to be switched. I tried different things but none of them workes as expected. Based on the first comment in this thread How to change Media Foundation Transform output frame(video) size? i started changing the implementation of SetOutputType. i called GetAttributeSize inside GetOutputType to receive the actual frame_size. It fails when i try to set a new frame_size (when starting playback i receive hresult 0xc00d36b4 (Data specified is invalid, inconsistent, or not supported by this object)

HRESULT CGrayscale::SetOutputType(
    DWORD           dwOutputStreamID,
    IMFMediaType    *pType, // Can be NULL to clear the output type.
    DWORD           dwFlags 
    )
{ ....
    //Receive the actual frame_size of pType (works as expected)
    hr = MFGetAttributeSize(
    pType,
    MF_MT_FRAME_SIZE,
    &width,
    &height
    ));
    ...
    //change the framesize 
    hr = MFSetAttributeSize(
    pType,
    MF_MT_FRAME_SIZE,
    height,
    width
    ));
}

I am sure i miss something here, so any hint will be greatly appreciated.

Thanks in advance

Answer 1

There is a transform available in W8+ that is supposed to do rotation. I haven't had much luck with it myself, but presumably it can be made to work. I'm going to assume that's not a viable solution for you.

The more interesting case is creating an MFT to do the transform.

It turns out there are a number of steps to turn 'Grayscale' into a rotator.

1) As you surmised, you need to affect the frame size on the output type. However, changing the type being passed to SetOutputType is just wrong. The pType being sent to SetOutputType is the type that the client is asking you to support. Changing that media type to something other than what they requested, then returning S_OK to say you support it makes no sense.

Instead what you need to change is the value sent back from GetOutputAvailableType.

2) When calculating the type to send back from GetOutputAvailableType, you need to base it on the IMFMediaType the client sent to SetInputType, with a few changes. And yes, you want to adjust MF_MT_FRAME_SIZE, but you probably also need to adjust MF_MT_DEFAULT_STRIDE, MF_MT_GEOMETRIC_APERTURE, and (possibly) MF_MT_MINIMUM_DISPLAY_APERTURE. Conceivably you might need to adjust MF_MT_SAMPLE_SIZE too.

3) You didn't say whether you intended the rotation amount to be fixed at start of stream, or something that varies during play. When I wrote this, I used the IMFAttributes returned from IMFTransform::GetAttributes to specify the rotation. Before each frame is processed, the current value is read. To make this work right, you need to be able to send MF_E_TRANSFORM_STREAM_CHANGE back from OnProcessOutput.

4) Being lazy, I didn't want to figure out how to rotate NV12 or YUY2 or some such. But there are functions readily available to do this for RGB32. So when my GetInputAvailableType is called, I ask for RGB32.

I experimented with supporting other input types, like RGB24, RGB565, etc, but ran into a problem. When your output type is RGB24, MF adds another MFT downstream to convert the RGB24 back into something it can more easily use (possibly RGB32). And that MFT doesn't support changing media types mid-stream. I was able to get this to work by accepting the variety of subtypes for input, but always outputting RGB32, rotated as specified.

This sounds complicated, but mostly it isn't. If you read the code you'd probably go "Oh, I get it." I'd offer you my source code, but I'm not sure how useful it would be for you. It's in c#, and you were asking about c++.

On the other hand, I'm making a template to make writing MFTs easier. ~A dozen lines of c# code to create the simplest possible MFT. The c# rotation MFT is ~131 lines as counted by VS's Analyze/Calculate code metrics (excluding the template). I'm experimenting with a c++ version, but it's still a bit rough.

Did I forget something? Probably a bunch of things. Like don't forget to generate a new Guid for your MFT instead of using Grayscale's. But I think I've hit the high points.

Edit: Now that my c++ version of the template is starting to work, I feel comfortable posting some actual code. This may make some of the points above clearer. For instance in #2, I talk about basing the output type on the input type. You can see that happening in CreateOutputFromInput. And the actual rotation code is in WriteIt().

I've simplified the code a bit for size, but hopefully this will get you to "Oh, I get it."

void OnProcessSample(IMFSample *pSample, bool Discontinuity, int InputMessageNumber)
{
    HRESULT hr = S_OK;

    int i = MFGetAttributeUINT32(GetAttributes(), AttribRotate, 0);
    i &= 7;

    // Will the output use different dimensions than the input?
    bool IsOdd = (i & 1) == 1;

    // Does the current AttribRotate rotation give a different 
    // orientation than the old one?
    if (IsOdd != m_WasOdd)
    {
        // Yes, change the output type.
        OutputSample(NULL, InputMessageNumber);
        m_WasOdd = IsOdd;
    }

    // Process it.
    DoWork(pSample, (RotateFlipType)i);

    // Send the modified input sample to the output sample queue.
    OutputSample(pSample, InputMessageNumber);
}

void OnSetInputType()
{
    HRESULT hr = S_OK;

    m_imageWidthInPixels = 0;
    m_imageHeightInPixels = 0;
    m_cbImageSize = 0;
    m_lInputStride = 0;

    IMFMediaType *pmt = GetInputType();

    // type can be null to clear
    if (pmt != NULL)
    {
        hr = MFGetAttributeSize(pmt, MF_MT_FRAME_SIZE, &m_imageWidthInPixels, &m_imageHeightInPixels);
        ThrowExceptionForHR(hr);

        hr = pmt->GetUINT32(MF_MT_DEFAULT_STRIDE, &m_lInputStride);
        ThrowExceptionForHR(hr);

        // Calculate the image size (not including padding)
        m_cbImageSize = m_imageHeightInPixels * m_lInputStride;
    }
    else
    {
        // Since the input must be set before the output, nulling the 
        // input must also clear the output.  Note that nulling the 
        // input is only valid if we are not actively streaming.

        SetOutputType(NULL);
    }
}

IMFMediaType *CreateOutputFromInput(IMFMediaType *inType)
{
    // For some MFTs, the output type is the same as the input type.  
    // However, since we are rotating, several attributes in the 
    // media type (like frame size) must be different on our output.  
    // This routine generates the appropriate output type for the 
    // current input type, given the current state of m_WasOdd.

    IMFMediaType *pOutputType = CloneMediaType(inType);

    if (m_WasOdd)
    {
        HRESULT hr;
        UINT32 h, w;

        // Intentionally backward
        hr = MFGetAttributeSize(inType, MF_MT_FRAME_SIZE, &h, &w);
        ThrowExceptionForHR(hr);

        hr = MFSetAttributeSize(pOutputType, MF_MT_FRAME_SIZE, w, h);
        ThrowExceptionForHR(hr);

        MFVideoArea *a = GetArea(inType, MF_MT_GEOMETRIC_APERTURE);
        if (a != NULL)
        {
            a->Area.cy = h;
            a->Area.cx = w;
            SetArea(pOutputType, MF_MT_GEOMETRIC_APERTURE, a);
        }

        a = GetArea(inType, MF_MT_MINIMUM_DISPLAY_APERTURE);
        if (a != NULL)
        {
            a->Area.cy = h;
            a->Area.cx = w;
            SetArea(pOutputType, MF_MT_MINIMUM_DISPLAY_APERTURE, a);
        }

        hr = pOutputType->SetUINT32(MF_MT_DEFAULT_STRIDE, w * 4);
        ThrowExceptionForHR(hr);
    }

    return pOutputType;
}

void WriteIt(BYTE *pBuffer, RotateFlipType fm)
{
    Bitmap *v = new Bitmap((int)m_imageWidthInPixels, (int)m_imageHeightInPixels, (int)m_lInputStride, PixelFormat32bppRGB, pBuffer);
    if (v == NULL)
        throw (HRESULT)E_OUTOFMEMORY;

    try
    {
        Status s;

        s = v->RotateFlip(fm);
        if (s != Ok)
            throw (HRESULT)E_UNEXPECTED;

        Rect r;

        if (!m_WasOdd)
        {
            r.Width = (int)m_imageWidthInPixels;
            r.Height = (int)m_imageHeightInPixels;
        }
        else
        {
            r.Height = (int)m_imageWidthInPixels;
            r.Width = (int)m_imageHeightInPixels;
        }

        BitmapData bmd;
        bmd.Width = r.Width,
        bmd.Height = r.Height,
        bmd.Stride = 4*bmd.Width;
        bmd.PixelFormat = PixelFormat32bppARGB; 
        bmd.Scan0 = (VOID*)pBuffer;
        bmd.Reserved = NULL;

        s = v->LockBits(&r, ImageLockModeRead + ImageLockModeUserInputBuf, PixelFormat32bppRGB, &bmd);
        if (s != Ok)
            throw (HRESULT)E_UNEXPECTED;

        s = v->UnlockBits(&bmd);
        if (s != Ok)
            throw (HRESULT)E_UNEXPECTED;
    }
    catch(...)
    {
        delete v;
        throw;
    }

    delete v;
}