How to sync AVPlayer and MTKView

Question

I have a project where users can take a video and later add filters to them or change basic settings like brightness and contrast. To accomplish this, I use BBMetalImage, which basically returns the video in a MTKView (named a BBMetalView in the project).

Everything works great - I can play the video, add filters and the desired effects, but there is no audio. I asked the author about this, who recommended using an AVPlayer (or AVAudioPlayer) for this. So I did. However, the video and audio are out of sync. Possibly because of different bitrates in the first place, and the author of the library also mentioned the frame rate can differ because of the filter process (the time this consumes is variable):

The render view FPS is not exactly the same to the actual rate. Because the video source output frame is processed by filters and the filter process time is variable.

First, I crop my video to the desired aspect ratio (4:5). I save this file (480x600) locally, using AVVideoProfileLevelH264HighAutoLevel as AVVideoProfileLevelKey. My audio configuration, using NextLevelSessionExporter, has the following setup: AVEncoderBitRateKey: 128000, AVNumberOfChannelsKey: 2, AVSampleRateKey: 44100.

Then, the BBMetalImage library takes this saved audio file and provides a MTKView (BBMetalView) to display the video, allowing me to add filters and effects in real time. The setup kind of looks like this:

self.metalView = BBMetalView(frame: CGRect(x: 0, y: self.view.center.y - ((UIScreen.main.bounds.width * 1.25) / 2), width: UIScreen.main.bounds.width, height: UIScreen.main.bounds.width * 1.25))
self.view.addSubview(self.metalView)
self.videoSource = BBMetalVideoSource(url: outputURL)
self.videoSource.playWithVideoRate = true
self.videoSource.audioConsumer = self.metalAudio
self.videoSource.add(consumer: self.metalView)
self.videoSource.add(consumer: self.videoWriter)
self.audioItem = AVPlayerItem(url: outputURL)                            
self.audioPlayer = AVPlayer(playerItem: self.audioItem)
self.playerLayer = AVPlayerLayer(player: self.audioPlayer)
self.videoPreview.layer.addSublayer(self.playerLayer!)
self.playerLayer?.frame = CGRect(x: 0, y: 0, width: 0, height: 0)
self.playerLayer?.backgroundColor = UIColor.black.cgColor
self.startVideo()

And startVideo() goes like this:

audioPlayer.seek(to: .zero)
audioPlayer.play()
videoSource.start(progress: { (frameTime) in
    print(frameTime)
}) { [weak self] (finish) in
guard let self = self else { return }
    self.startVideo()
}

This is all probably pretty vague because of the external library/libraries. However, my question is pretty simple: is there any way I can sync the MTKView with my AVPlayer? It would help me a lot and I'm sure Silence-GitHub would also implement this feature into the library to help a lot of other users. Any ideas on how to approach this are welcome!

Answer 1

I custom the BBMetalVideoSource as follow then it worked:

1. Create a delegate in BBMetalVideoSource to get the current time of the audio player with which we want to sync

2. In func private func processAsset(progress:, completion:), I replace this block of code if useVideoRate { //... } by:

   if useVideoRate {
       if let playerTime = delegate.getAudioPlayerCurrentTime() {
           let diff = CMTimeGetSeconds(sampleFrameTime) - playerTime
           if diff > 0.0 {
               sleepTime = diff
               if sleepTime > 1.0 {
                   sleepTime = 0.0
               }
               usleep(UInt32(1000000 * sleepTime))
           } else {
               sleepTime = 0
           }
       }
   }
   

This code help us resolve both problems: 1. No audio when preview video effect, and 2. Sync audio with video.