Decoding H264 VideoToolkit API fails with Error -8971 in VTDecompressionSessionCreate

Question

I am trying to write a Video decoder using the Hardware supported Video Toolkit Decoder. But if I try to initialize the decoding session like in the example posted below, I get the error -8971 while calling VTDecompressionSessionCreate. Can anyone tell me what I am doing wrong here?

Thank you and best regards,

Oliver

OSStatus status;

int tmpWidth = sps.EncodedWidth();
int tmpHeight = sps.EncodedHeight();
NSLog(@"Got new Width and Height from SPS - %dx%d", tmpWidth, tmpHeight);

const VTDecompressionOutputCallbackRecord callback = { ReceivedDecompressedFrame, self };
status = CMVideoFormatDescriptionCreate(NULL,
                                       kCMVideoCodecType_H264,
                                       tmpWidth,
                                       tmpHeight,
                                       NULL,
                                       &decoderFormatDescription);

if (status == noErr)
{
    // Set the pixel attributes for the destination buffer
    CFMutableDictionaryRef destinationPixelBufferAttributes = CFDictionaryCreateMutable(
                                                                 NULL, // CFAllocatorRef allocator
                                                                 0,    // CFIndex capacity
                                                                 &kCFTypeDictionaryKeyCallBacks, 
                                                                 &kCFTypeDictionaryValueCallBacks);

    SInt32 destinationPixelType = kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange;
    CFDictionarySetValue(destinationPixelBufferAttributes,kCVPixelBufferPixelFormatTypeKey, CFNumberCreate(NULL, kCFNumberSInt32Type, &destinationPixelType));
    CFDictionarySetValue(destinationPixelBufferAttributes,kCVPixelBufferWidthKey, CFNumberCreate(NULL, kCFNumberSInt32Type, &tmpWidth));
    CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferHeightKey, CFNumberCreate(NULL, kCFNumberSInt32Type, &tmpHeight));
    CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferOpenGLCompatibilityKey, kCFBooleanTrue);

    // Set the Decoder Parameters
    CFMutableDictionaryRef decoderParameters = CFDictionaryCreateMutable(
                                                        NULL, // CFAllocatorRef allocator
                                                        0,    // CFIndex capacity
                                                        &kCFTypeDictionaryKeyCallBacks,
                                                        &kCFTypeDictionaryValueCallBacks);

    CFDictionarySetValue(decoderParameters,kVTDecompressionPropertyKey_RealTime, kCFBooleanTrue);

    // Create the decompression session
    // Throws Error -8971 (codecExtensionNotFoundErr)
    status = VTDecompressionSessionCreate(NULL, decoderFormatDescription, decoderParameters, destinationPixelBufferAttributes, &callback, &decoderDecompressionSession);

    // release the dictionaries
    CFRelease(destinationPixelBufferAttributes);
    CFRelease(decoderParameters);

    // Check the Status
    if(status != noErr)
    {
        NSLog(@"Error %d while creating Video Decompression Session.", (int)status);
        continue;
    }
}
else
{
    NSLog(@"Error %d while creating Video Format Descripttion.", (int)status);
    continue;
}

Answer 1

I also stumbled with kVTVideoDecoderBadDataErr. In my case I was changing the header 0x00000001 with the size of the NAL package which included the 4 bytes of this header, that was the reason. I changed the size to not include these 4 bytes (frame_size = sizeof(NAL) - 4). This size should be encoded in big-endian.

Answer 2

You need to create the CMFormatDescriptionRef from your SPS and PPS like

CMFormatDescriptionRef decoderFormatDescription;
const uint8_t* const parameterSetPointers[2] = { (const uint8_t*)[currentSps bytes], (const uint8_t*)[currentPps bytes] };
const size_t parameterSetSizes[2] = { [currentSps length], [currentPps length] };
status = CMVideoFormatDescriptionCreateFromH264ParameterSets(NULL,
                                                             2,
                                                             parameterSetPointers,
                                                             parameterSetSizes,
                                                             4,
                                                             &decoderFormatDescription);

Also if you are getting your Video Data in Annex-B format you need to remove the start code and replace it with the 4-Byte size information for the decoder to recognize it as avcc formated (Thats what the 5th parameter to CMVideoFormatDescriptionCreateFromH264ParameterSets is for).