AVAssetImageGenerator returns sometimes same image from 2 successive frames

Question

I'm currently extracting every frame from a video with AVAssetImageGenerator, but sometimes it returns me successively 2 times almost the same image (they do not have the same "frame time"). The funny thing is it always happen (in my test video) each 5 frames.

Here and here are the two images (open each in new tab then switch the tabs to see the differences).

Here's my code :

//setting up generator & compositor
self.generator = [AVAssetImageGenerator assetImageGeneratorWithAsset:asset];
generator.appliesPreferredTrackTransform = YES;
self.composition = [AVVideoComposition videoCompositionWithPropertiesOfAsset:asset];

NSTimeInterval duration = CMTimeGetSeconds(asset.duration);
NSTimeInterval frameDuration = CMTimeGetSeconds(composition.frameDuration);
CGFloat totalFrames = round(duration/frameDuration);

NSMutableArray * times = [NSMutableArray array];
for (int i=0; i<totalFrames; i++) {
    NSValue * time = [NSValue valueWithCMTime:CMTimeMakeWithSeconds(i*frameDuration, composition.frameDuration.timescale)];
    [times addObject:time];
}

AVAssetImageGeneratorCompletionHandler handler = ^(CMTime requestedTime, CGImageRef im, CMTime actualTime, AVAssetImageGeneratorResult result, NSError *error){
    // If actualTime is not equal to requestedTime image is ignored
    if(CMTimeCompare(actualTime, requestedTime) == 0) {
        if (result == AVAssetImageGeneratorSucceeded) {
            NSLog(@"%.02f     %.02f", CMTimeGetSeconds(requestedTime), CMTimeGetSeconds(actualTime));
            // Each log have differents actualTimes.
            // frame extraction is here...
        }
    }
};

generator.requestedTimeToleranceBefore = kCMTimeZero;
generator.requestedTimeToleranceAfter = kCMTimeZero;
[generator generateCGImagesAsynchronouslyForTimes:times completionHandler:handler];

Any idea where it could come from?

Answer 1

Please see the following properties of AVAssetImageGenerator. You should set kCMTimeZero for both properties to get the exact frames.

/* The actual time of the generated images will be within the range [requestedTime-toleranceBefore, requestedTime+toleranceAfter] and may differ from the requested time for efficiency.
   Pass kCMTimeZero for both toleranceBefore and toleranceAfter to request frame-accurate image generation; this may incur additional decoding delay.
   Default is kCMTimePositiveInfinity. */
@property (nonatomic) CMTime requestedTimeToleranceBefore NS_AVAILABLE(10_7, 5_0);
@property (nonatomic) CMTime requestedTimeToleranceAfter NS_AVAILABLE(10_7, 5_0);

Before I set kCMTimeZero for both properties, I got some same images for different request time as you experienced. Just try the following code.

self.imageGenerator = [AVAssetImageGenerator assetImageGeneratorWithAsset:myAsset];
self.imageGenerator.requestedTimeToleranceBefore = kCMTimeZero;
self.imageGenerator.requestedTimeToleranceAfter = kCMTimeZero;

Answer 2

I was having the same issue of you, but much more evident, the duplication was happening when the interval between the two frames was under 1.0 second and I realised it was depending on the timescale I was using for generating CMTime values.

Before

CMTime requestTime = CMTimeMakeWithSeconds(imageTime, 1);

After

CMTime requestTime = CMTimeMakeWithSeconds(imageTime, playerItem.asset.duration.timescale);

... and Boom, no more duplication :)

So maybe you can try to increase, double perhaps, the timescale, using your code:

NSValue * time = [NSValue valueWithCMTime:CMTimeMakeWithSeconds(i*frameDuration, composition.frameDuration.timescale*2)]; // *2 at the end

For future references here is my code:

    playerItem = [AVPlayerItem playerItemWithURL:item.movieUrl];
    imageGenerator = [[AVAssetImageGenerator alloc] initWithAsset:playerItem.asset];
    imageGenerator.requestedTimeToleranceAfter = kCMTimeZero;
    imageGenerator.requestedTimeToleranceBefore = kCMTimeZero;

    float duration = item.duration;
    float interval = item.interval;

    NSLog(@"\nItem info:\n%f \n%f", duration,interval);

    NSString *srcPath = nil;
    NSString *zipPath = nil;

    srcPath = [item.path stringByAppendingPathComponent:@"info.json"];
    zipPath = [NSString stringWithFormat:@"/%@/info.json",galleryID];

    [zip addFileToZip:srcPath newname:zipPath level:0];

    NSTimeInterval frameNum = item.duration / item.interval;
    for (int i=0; i<=frameNum; i++)
    {
        NSArray* cachePathArray = NSSearchPathForDirectoriesInDomains(NSCachesDirectory, NSUserDomainMask, YES);
        NSString* cachePath = [cachePathArray lastObject];

        srcPath = [cachePath stringByAppendingPathComponent:@"export-tmp.jpg"];
        zipPath = [NSString stringWithFormat:@"/%@/%d.jpg",galleryID,i];

        float imageTime = ( i * interval );

        NSError *error = nil;
        CMTime requestTime = CMTimeMakeWithSeconds(imageTime, playerItem.asset.duration.timescale);
        CMTime actualTime;

        CGImageRef imageRef = [imageGenerator copyCGImageAtTime:requestTime actualTime:&actualTime error:&error];

        if (error == nil) {
            float req = ((float)requestTime.value/requestTime.timescale);
            float real = ((float)actualTime.value/actualTime.timescale);
            float diff = fabsf(req-real);

            NSLog(@"copyCGImageAtTime: %.2f, %.2f, %f",req,real,diff);
        }
        else
        {
            NSLog(@"copyCGImageAtTime: error: %@",error.localizedDescription);
        }



        // consider using CGImageDestination -> http://stackoverflow.com/questions/1320988/saving-cgimageref-to-a-png-file
        UIImage *img = [UIImage imageWithCGImage:imageRef];
        CGImageRelease(imageRef);  // CGImageRef won't be released by ARC



        [UIImageJPEGRepresentation(img, 100) writeToFile:srcPath atomically:YES];

        if (srcPath != nil && zipPath!= nil)
        {
            [zip addFileToZip:srcPath newname:zipPath level:0]; // 0 = no compression. everything is a jpg image
            unlink([srcPath UTF8String]);
        }