AVFoundation record 10 bit HDR video on iPhone 12

Question

iPhone 12/12 pro supports recording Dolby vision HDR video in 10 bit format instead of 8 bits but it is not clear from iOS 14.1 SDK if AVCaptureVideoDataOutput supports delivery of 10 bit sample buffers that can be appended to video file using AVAssetWriter. Has anyone figured it out whether it is possible or not in SDK?

EDIT: A number of apps such as Apple's Clips app have started supporting Dolby Vision 10 bit video recording. But I tried every available API including videoHDREnabled but it doesn't work. So the clear question is how to record HDR (Dolby vision) video using AVFoundation APIs?

EDIT2: I was able to figure out device formats that support 10 bit pixel buffer formats (which is 'x420', NOT the ones with 420v or 420f as media subtype) . On iPhone 12 mini, 4 device formats support 10 bit pixel buffer delivery in kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, even though AVFoundation documentation says this is not a supported pixel format (Quoting - "On iOS, the only supported key is kCVPixelBufferPixelFormatTypeKey. Supported pixel formats are kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange and kCVPixelFormatType_32BGRA."). Next step is to identify if HDR format for recording can be manually chosen to be Dolby Vision, HLG, or HDR10.

Answer 1

Ok none of the answers given were correct, so I researched after grabbing an iPhone 12 mini in hand and this is what I found.

AVFoundation documentation is silent and even incorrect at times. One can infer from the documentation that it is not possible to get 10 bit HDR sample buffers, specifically if one reads the documentation of videoSettings property of AVCaptureVideoDataOutput:

   On iOS, the only supported key is kCVPixelBufferPixelFormatTypeKey. 
   Supported pixel formats are kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, 
   kCVPixelFormatType_420YpCbCr8BiPlanarFullRange and kCVPixelFormatType_32BGRA

It appears from the documentation that one could never get 10 bit frames. But on probing -[AVCaptureDevice formats], one can find 4 formats that are different and have mediaSubtype as 'x420', which is kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, a 10 bit format. The moment -[AVCaptureDevice activeFormat] is set to one of these 4 formats, AVCaptureVideoDataOutput changes the sample buffer format to kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange! Active Color Space of AVCaptureDevice also changes to AVCaptureColorSpace_HLG_BT2020.

Answer 2

Nov 26 update.

As @Deepak posted in his own answer and comments, 'x420' tagged format will let the camera work in HLG mode. All the available HLG enabled formats from IP12 Pro are updated below.

Original answer

For iOS 14.2, I can dump all the available formats from AVCaptureDevice instance, seems the log output fairly explains itself. As commented below, hopefully setting the AVCaptureDevice.activeFormat to one of the HDR+wide color format will do the job.

<AVCaptureDeviceFormat: 0x282d8daf0 'vide'/'x420' 1280x 720, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @2.91), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth>
<AVCaptureDeviceFormat: 0x282d8dac0 'vide'/'x420' 1280x 720, { 1- 60 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @2.91), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth>
<AVCaptureDeviceFormat: 0x282d8da50 'vide'/'x420' 1920x1080, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @1.94), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth, supports multicam>
<AVCaptureDeviceFormat: 0x282d8da30 'vide'/'x420' 1920x1080, { 1- 60 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @1.94), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports multicam>
<AVCaptureDeviceFormat: 0x282d8d9e0 'vide'/'x420' 1920x1440, { 1- 30 fps}, HRSI:4032x3024, fov:67.096, max zoom:189.00 (upscales @2.10), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth, supports multicam>
<AVCaptureDeviceFormat: 0x282d8d950 'vide'/'x420' 3840x2160, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:125.25 (upscales @1.00), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports multicam>

As of Nov 23, it is still an ongoing investigation, I think some joint effort is needed, or some Apple engineer can have a look at this.

I believe I have watched all the available WWDC17/18/19/20 sessions on this topic, and with the new iPhone 12 release, some findings here.

Capturing HDR from camera and saved as 10 bit HLG video directly is only capable on iPhone 12 and newer. This is what it claims in the product release, and I have got sample videos from my friend's new phone, it is working as expected.

In WWDC2020, Export HDR media in your app with AVFoundation , it claims:

At this point, I’d like to briefly touch on which Apple platforms can support HDR export.

iOS supports HEVC hardware encoding on devices with Apple A10 Fusion chips or newer.

Fortunately A10 devices have been around for a while, dating back to the iPhone 7, iPads released in 2018, and the 2019 iPod touch.

In regards to Macs, both HEVC and Apple ProRes software encoders are available on all Macs.

HEVC hardware encoding is generally available on 2017 and newer Macs running the new macOS.

Hardware encoding will make the export significantly faster.

Also in this video, it claims HDR export only works 10bit HEVC encoding, so the A10+ SoC should have 10bit HEVC encoding capability. This is a guess, I can edit the iPhone12 HLG video within official Photo app on iPhone 11 and SE2, and the writing performance(4k@60p, HLG) is quite good, which is a good indicator. However, I have no luck to make this work in code, the sample code listed in the video could not be the full picture, and I am having difficulty to find a working demo yet. And in theory, older devices should have the capability to record 10bit HLG also, or the camera, thermal/power budget is the limitation here.

However, the only relevant HDR key in this is the VideoProfileLevelKey which must be set to HEVC_Main10_AutoLevel when exporting HDR using the HEVC codec.

Note that 8-bit HEVC HDR is not supported, and this key is not applicable to ProRes exports.

All right, let’s take the time now to summarize how you would configure the keys I just discussed when outputting to two common HDR formats: HLG and HDR10. This table shows what the relevant HDR settings are for exporting an HLG file.

Another video worth watching again and again: Edit and play back HDR video with AVFoundation

During the test, I do get a CVPixelBuffer(format:kCVPixelFormatType_420YpCbCr10BiPlanarFullRange) that is HDR enabled, and correctly color managed from sample HLG video. This is a dump from my console log, and it is working on any iOS 14 enabled devices, even with the quite old iPhone6s(A9), because it only involves 10bit HEVC decoding here.

_displayLinkDidRefresh():121 - Optional(<CVPixelBuffer 0x281300500 width=3840 height=2160 pixelFormat=xf20 iosurface=0x282008050 planes=2 poolName=450:decode_1>
<Plane 0 width=3840 height=2160 bytesPerRow=7680>
<Plane 1 width=1920 height=1080 bytesPerRow=7680>
<attributes={
    PixelFormatDescription =     {
        BitsPerComponent = 10;
        CGBitmapContextCompatibility = 0;
        CGImageCompatibility = 0;
        ComponentRange = FullRange;
        ContainsAlpha = 0;
        ContainsGrayscale = 0;
        ContainsRGB = 0;
        ContainsYCbCr = 1;
        FillExtendedPixelsCallback = {length = 24, bytes = 0x0000000000000000b48ab8a1010000000000000000000000};
        IOSurfaceCoreAnimationCompatibility = 1;
        IOSurfaceCoreAnimationCompatibilityHTPCOK = 1;
        IOSurfaceOpenGLESTextureCompatibility = 1;
        OpenGLESCompatibility = 1;
        PixelFormat = 2019963440;
        Planes =         (
                        {
                BitsPerBlock = 16;
                HorizontalSubsampling = 1;
                VerticalSubsampling = 1;
            },
                        {
                BitsPerBlock = 32;
                BlackBlock = {length = 4, bytes = 0x00800080};
                HorizontalSubsampling = 2;
                VerticalSubsampling = 2;
            }
        );
    };
} propagatedAttachments={
    CVFieldCount = 1;
    CVImageBufferChromaLocationBottomField = Left;
    CVImageBufferChromaLocationTopField = Left;
    CVImageBufferColorPrimaries = "ITU_R_2020";
    CVImageBufferTransferFunction = "ITU_R_2100_HLG";
    CVImageBufferYCbCrMatrix = "ITU_R_2020";
    QTMovieTime =     {
        TimeScale = 600;
        TimeValue = 12090;
    };
} nonPropagatedAttachments={
}>)

Answer 3

Forcing a format with BT2020 is the correct way to ensure you're shooting in Dolby Vision. iOS 14.1 or high is required to do this. Here's an abbreviated snippet of how I'm doing this:

// Setup your session
session.beginConfiguration()
session.sessionPreset = .hd1280x720

// Add your camera to the session
let camera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front)
let cameraInput = try AVCaptureDeviceInput(device: camera)
session.addInput(cameraInput)

// Important! Commit the session configuration before configuring your camera
session.commitConfiguration()

// Configure camera
try camera.lockForConfiguration()

// Force HDR on
camera.automaticallyAdjustsVideoHDREnabled = false
camera.isVideoHDREnabled = true

// Find the first 720p format that supports the correct colorspace
let desiredColorSpace = AVCaptureColorSpace.HLG_BT2020
let desiredFormat = camera.formats.first { format in
    // You could of course choose a different resolution if desired
    format.formatDescription.dimensions == CMVideoDimensions(width: 1280, height: 720) &&
        format.supportedColorSpaces.contains(desiredColorSpace)
}

// Set the HDR format
if let format = desiredFormat {
    camera.activeFormat = format
    camera.activeColorSpace = desiredColorSpace
} else {
    assertionFailure("Counldn't find HDR camera format")
}

camera.unlockForConfiguration()