App Crashes while getting all the Frames from a video using AVAssetImageGenerator for long videos

Question

I have successfully extracted all the frames from a video and it is working fine for smaller videos but when I try to extract frames from a video more than 60 secs the app crashes on device.

I am extracting 30 frames per second from the video.

Following is the code I have written :-

var  videoSec = Float64(0)

 func startImageConversion(){
        let filePath = NSURL.fileURLWithPath(self.savedVideoURL)
        videoSec = self.getVideoTime(filePath)
        videoImagesArray =  self.getImagesArrayFromVideo(filePath)
        print("Images Count \(videoImagesArray.count)")
    }

// MARK: - Video Editor Functions

    func getImagesArrayFromVideo(filePath:NSURL) -> NSMutableArray
    {
        let imageArray = NSMutableArray()
        print("Video Sec is ",videoSec)
        let vidSec = Float64(videoSec)

        let theOpts = [
            AVURLAssetPreferPreciseDurationAndTimingKey : true,
            AVURLAssetReferenceRestrictionsKey : 0 // AVAssetReferenceRestrictions.RestrictionForbidNone
        ]
        let asset = AVURLAsset(URL: filePath, options: theOpts)
        let generator = AVAssetImageGenerator(asset: asset)
        generator.maximumSize = CGSize(width: Double(self.view.frame.size.width),
                                       height: Double(self.view.frame.size.height))

        generator.appliesPreferredTrackTransform = false
        generator.requestedTimeToleranceBefore = kCMTimeZero
        generator.requestedTimeToleranceAfter = kCMTimeZero
        let vid_length:CMTime = asset.duration
        let fps = vid_length.timescale
        print("Video Lenght :- \(vid_length) FPS is :- \(fps)")


        let mainValue = Float64(vid_length.value)

        let divide = vidSec*30
        let byVal = mainValue/divide


        for i in 0.stride(through: Float64(vid_length.value), by: byVal)
        {

            var image:CGImage!//UIImage()
            let divident = Float64(i)
//            let mileSec = Float64(divident / 1000)
//            print(Sec)
            image  = self.generateVideoThumbs(filePath, second: divident,
                                              thumbWidth: Double(self.view.frame.size.width),
                                              generator : generator,
                                              fps : fps
            )

            if image != nil {

                imageArray.addObject(image)

            }
        }

        print("value of Array is ",imageArray.count)

        return imageArray

    }



 private func getVideoTime(url: NSURL) -> Float64
    {
        let videoTime = AVURLAsset(URL: url, options: nil)
        print("videoTime.preferredRate = \(videoTime.preferredRate)")
        return CMTimeGetSeconds(videoTime.duration)
    }



 private func generateVideoThumbs(url: NSURL, second: Float64, thumbWidth: Double, generator:AVAssetImageGenerator, fps: CMTimeScale) -> CGImage! {
        let thumbTime = CMTimeMake(Int64(second), fps)
        var actualTime : CMTime = CMTimeMake(0, 0)
        print("thumbTime - \(thumbTime)")
        do {
            let ref = try generator.copyCGImageAtTime(thumbTime, actualTime: &actualTime)
            print("actualTime - \(actualTime)")
            return ref//UIImage(CGImage: ref)
        }catch {
            print(error)
            return nil
        }
     }

Everything works fine if the video is less than 60 secs or so. Also it works fine on simulator but disconnects the device without any warning or error. Any Help will appreciated, Thanks

Answer 1

autoreleasepool{
if image != nil {
var newImage:UIImage = UIImage(CGImage: cgImage)
                imageArray.addObject(image)
            }
}

As in the above answer dive mentioned its the problem with memory. Just use this code. I have tested it and it worked fine.

Answer 2

Core Graphics doesn't support autorelease pool in Objective C, but Swift ARC can handle CF types, anyway you can still have issues with CGImage releasing in some cases, maybe it's the case and you obviously have an issue with the size of your imageArray and I'm sure that your device is disconnected because of out of memory SIGTERM. iOS Simulator use shared memory on your modern 15" develop MacBook, so, everything will work there.

I suggest you to rewrite your generateVideoThumbs(...) function and use UIImage as return value (you've tried it already as I can see, just return your solution with UIImage):

private func generateVideoThumbs(url: NSURL, second: Float64, thumbWidth: Double, generator:AVAssetImageGenerator, fps: CMTimeScale) -> UIImage! {
        let thumbTime = CMTimeMake(Int64(second), fps)
        var actualTime : CMTime = CMTimeMake(0, 0)
        print("thumbTime - \(thumbTime)")
        do {
            let ref = try generator.copyCGImageAtTime(thumbTime, actualTime: &actualTime)
            let resultImage = UIImage.init(ref)
            print("actualTime - \(actualTime)")
            return resultImage
        }catch {
            print(error)
            return nil
        }
     }

And second, I suggest you to save your images to disk and store only links in your imageArray. Something like this:

if image != nil {
    let data = UIImagePNGRepresentation(image)
    let filename = NSTemporaryDirectory().appendingPathComponent("\(i).png")
    try? data.write(to: filename)
    imageArray.addObject(filename)
}

Then you can extract your images from URLs and do whatever you plan to do with them. It allows you to avoid memory pressure and use your array in any way you want (copy, forward to another class, etc).

P.S. I wrote the code without compilation, but I think that idea is clear.