Maintaining good scroll performance when using AVPlayer

Question

I'm working on an application where there is a collection view, and cells of the collection view can contain video. Right now I'm displaying the video using AVPlayer and AVPlayerLayer. Unfortunately, the scrolling performance is terrible. It seems like AVPlayer, AVPlayerItem, and AVPlayerLayer do a lot of their work on the main thread. They are constantly taking out locks, waiting on semaphores, etc. which is blocking the main thread and causing severe frame drops.

Is there any way to tell AVPlayer to stop doing so many things on the main thread? So far nothing I've tried has solved the problem.

I also tried building a simple video player using AVSampleBufferDisplayLayer. Using that I can make sure that everything happens off the main thread, and I can achieve ~60fps while scrolling and playing video. Unfortunately that method is much lower level, and it doesn't provide things like audio playback and time scrubbing out of the box. Is there any way to get similar performance with AVPlayer? I'd much rather use that.

Edit: After looking into this more, it doesn't look like it's possible to achieve good scrolling performance when using AVPlayer. Creating an AVPlayer and associating in with an AVPlayerItem instance kicks off a bunch of work which trampolines onto the main thread where it then waits on semaphores and tries to acquire a bunch of locks. The amount of time this stalls the main thread increases quite dramatically as the number of videos in the scrollview increases.

AVPlayer dealloc also seems to be a huge problem. Dealloc'ing an AVPlayer also tries to synchronize a bunch of stuff. Again, this gets extremely bad as you create more players.

This is pretty depressing, and it makes AVPlayer almost unusable for what I'm trying to do. Blocking the main thread like this is such an amateur thing to do so it's hard to believe Apple engineers would've made this kind of mistake. Anyways, hopefully they can fix this soon.

Answer 1

Build your AVPlayerItem in a background queue as much as possible (some operations you have to do on the main thread, but you can do setup operations and waiting for video properties to load on background queues - read the docs very carefully). This involves voodoo dances with KVO and is really not fun.

The hiccups happen while the AVPlayer is waiting for the AVPlayerItems status to become AVPlayerItemStatusReadyToPlay. To reduce the length of the hiccups you want to do as much as you can to bring the AVPlayerItem closer to AVPlayerItemStatusReadyToPlay on a background thread before assigning it to the AVPlayer.

It's been a while since I actually implemented this, but IIRC the main thread blocks are caused because the underlying AVURLAsset's properties are lazy-loaded, and if you don't load them yourself, they get busy-loaded on the main thread when the AVPlayer wants to play.

Check out the AVAsset documentation, especially the stuff around AVAsynchronousKeyValueLoading. I think we needed to load the values for duration and tracks before using the asset on an AVPlayer to minimize the main thread blocks. It's possible we also had to walk through each of the tracks and do AVAsynchronousKeyValueLoading on each of the segments, but I don't remember 100%.

Answer 2

Don't know if this will help – but here's some code I'm using to load videos on background queue that definitely helps with main thread blocking (Apologies if it doesn't compile 1:1, I abstracted from a larger code base I'm working on):

func loadSource() {     self.status = .Unknown      let operation = NSBlockOperation()     operation.addExecutionBlock { () -> Void in     // create the asset     let asset = AVURLAsset(URL: self.mediaUrl, options: nil)     // load values for track keys     let keys = ["tracks", "duration"]     asset.loadValuesAsynchronouslyForKeys(keys, completionHandler: { () -> Void in         // Loop through and check to make sure keys loaded         var keyStatusError: NSError?         for key in keys {             var error: NSError?             let keyStatus: AVKeyValueStatus = asset.statusOfValueForKey(key, error: &error)             if keyStatus == .Failed {                 let userInfo = [NSUnderlyingErrorKey : key]                 keyStatusError = NSError(domain: MovieSourceErrorDomain, code: MovieSourceAssetFailedToLoadKeyValueErrorCode, userInfo: userInfo)                 println("Failed to load key: \(key), error: \(error)")             }             else if keyStatus != .Loaded {                 println("Warning: Ignoring key status: \(keyStatus), for key: \(key), error: \(error)")             }         }         if keyStatusError == nil {             if operation.cancelled == false {                 let composition = self.createCompositionFromAsset(asset)                 // register notifications                 let playerItem = AVPlayerItem(asset: composition)                 self.registerNotificationsForItem(playerItem)                 self.playerItem = playerItem                 // create the player                 let player = AVPlayer(playerItem: playerItem)                 self.player = player             }         }         else {             println("Failed to load asset: \(keyStatusError)")         }     })      // add operation to the queue     SomeBackgroundQueue.addOperation(operation) }  func createCompositionFromAsset(asset: AVAsset, repeatCount: UInt8 = 16) -> AVMutableComposition {      let composition = AVMutableComposition()      let timescale = asset.duration.timescale      let duration = asset.duration.value      let editRange = CMTimeRangeMake(CMTimeMake(0, timescale), CMTimeMake(duration, timescale))      var error: NSError?      let success = composition.insertTimeRange(editRange, ofAsset: asset, atTime: composition.duration, error: &error)      if success {          for _ in 0 ..< repeatCount - 1 {           composition.insertTimeRange(editRange, ofAsset: asset, atTime: composition.duration, error: &error)          }      }      return composition }