AVCaptureSession video stabilization lag

Question

Previously I was adding an AVCaptureMovieFileOutput to my AVCaptureSession and enabling cinematic preferredVideoStabilizationMode, receiving great, expected results.

I just switched over to an AVCaptureVideoDataOutput and the cinematic stabilization is causing extremely noticeable lag — about a full second delay before the image preview updates.

Setup:

//  Video data output
let outputSettings: [String: Any] = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]
videoDataOutput = AVCaptureVideoDataOutput()
videoDataOutput?.videoSettings = outputSettings
let videoSessionQueue = DispatchQueue(label: "video_capture_session_queue")
videoDataOutput?.setSampleBufferDelegate(self, queue: videoSessionQueue)
videoDataOutput?.alwaysDiscardsLateVideoFrames = true

//  Audio data output
let audioDataOutput = AVCaptureAudioDataOutput()
let audioSessionQueue = DispatchQueue(label: "audio_capture_session_queue")
audioDataOutput.setSampleBufferDelegate(self, queue: audioSessionQueue)

//  Configure session
captureSession.beginConfiguration()

let videoInput = try AVCaptureDeviceInput(device: currentCameraDevice)
captureSession.addInput(videoInput)
captureSession.addOutput(videoDataOutput!)

if let videoDataOutputConnection = videoDataOutput?.connection(with: .video), videoDataOutputConnection.isVideoStabilizationSupported {
    videoDataOutputConnection.preferredVideoStabilizationMode = .cinematic
}

if let audioInput = AVCaptureDevice.default(for: .audio) {
    try captureSession.addInput(AVCaptureDeviceInput(device: audioInput))
    captureSession.addOutput(audioDataOutput)
}

captureSession.commitConfiguration()

captureSession.startRunning()

Does anyone have any insight or a solution? Thanks in advance.

Answer 1

To anyone interested, here's what I've learned after a few more days of digging:

The lag can't be avoided when rendering to a non-AVCaptureVideoPreviewLayer — which is essentially any time you're running an AVCaptureVideoDataOutput to process individual frames (and true for both CI and Metal setups).

Stabilization mode .cinematic leads to the most noticeable lag — ~1 second. Switching to .standard cuts that down to ~1/10 of a second, and produces better results than no stabilization at all.