How to merge a *Single Image* with a video

Question

I am trying to combine a single video with a single image. This is not trying to combine many images into a single video such as

• create movie from [UIImage], Swift

• How do I export UIImage array as a movie?

• Make movie file with picture Array and song file, using AVAsset

I am using AVMutableComposition to combine tracks. My app has the ability to combine videos and images (but as it stands, the videos combining is fine!) I attempt to use AVAssetWriter to turn a single image into a video (I believe this is where my problem is but not 100% sure). Then I save that to the app (documents directory). From there, I access it inside my merger, and combine a video and the image that now has turned into the video.

Flow:

User selects image ->

Image into AVAssetWriter to change to video ->

Merge a video I already have preset with the video ->

Result: Make 1 video from the image selected and preset video.

The problem with what I have: My code is providing a blank space where the image inside the video should be. As in, the ImageConverter file I have, will convert it to video, but I will only SEE the very LAST frame as the image, while every other frame is transparent, as if the picture is not there. So if I convert the image to a video for 5 seconds (let's say at 30 frames/sec) then I will see blank space for (30*5)-1 frames and then that last frame, the picture will finally appear. I am just looking for guidance on how to make a single image into a video OR combine a video and an image together WITHOUT converting the image into a video. Thanks!

Merge Files here

func merge() {
    if let firstAsset = controller.firstAsset, secondAsset = self.asset {

        // 1 - Create AVMutableComposition object. This object will hold your AVMutableCompositionTrack instances.
        let mixComposition = AVMutableComposition()

        let firstTrack = mixComposition.addMutableTrackWithMediaType(AVMediaTypeVideo,
                                                                     preferredTrackID: Int32(kCMPersistentTrackID_Invalid))
        do {
            try firstTrack.insertTimeRange(CMTimeRangeMake(kCMTimeZero, CMTime(seconds: 8, preferredTimescale: 600)),
                                           ofTrack: firstAsset.tracksWithMediaType(AVMediaTypeVideo)[0] ,
                                           atTime: kCMTimeZero)
        } catch _ {
            print("Failed to load first track")
        }

        do {
            //HERE THE TIME IS 0.666667, BUT SHOULD BE 0
            print(CMTimeGetSeconds(secondAsset.duration), CMTimeGetSeconds(firstTrack.timeRange.duration))
            try firstTrack.insertTimeRange(CMTimeRangeMake(kCMTimeZero, secondAsset.duration),
                                            ofTrack: secondAsset.tracksWithMediaType(AVMediaTypeVideo)[0],
                                            atTime: firstTrack.timeRange.duration)
        } catch _ {
            print("Failed to load second track")
        }
        do {
            try firstTrack.insertTimeRange(CMTimeRangeMake(CMTime(seconds: 8+CMTimeGetSeconds(secondAsset.duration), preferredTimescale: 600), firstAsset.duration),
                                           ofTrack: firstAsset.tracksWithMediaType(AVMediaTypeVideo)[0] ,
                                           atTime: firstTrack.timeRange.duration+secondTrack.timeRange.duration)
        } catch _ {
            print("failed")
        }

        // 3 - Audio track
        if let loadedAudioAsset = controller.audioAsset {
            let audioTrack = mixComposition.addMutableTrackWithMediaType(AVMediaTypeAudio, preferredTrackID: 0)
            do {
                try audioTrack.insertTimeRange(CMTimeRangeMake(kCMTimeZero, firstAsset.duration),
                                               ofTrack: loadedAudioAsset.tracksWithMediaType(AVMediaTypeAudio)[0] ,
                                               atTime: kCMTimeZero)
            } catch _ {
                print("Failed to load Audio track")
            }
        }

        // 4 - Get path
        let documentDirectory = NSSearchPathForDirectoriesInDomains(.DocumentDirectory, .UserDomainMask, true)[0]
        let dateFormatter = NSDateFormatter()
        dateFormatter.dateStyle = .LongStyle
        dateFormatter.timeStyle = .ShortStyle
        let date = dateFormatter.stringFromDate(NSDate())
        let savePath = (documentDirectory as NSString).stringByAppendingPathComponent("mergeVideo.mov")
        let url = NSURL(fileURLWithPath: savePath)
        _ = try? NSFileManager().removeItemAtURL(url)

        // 5 - Create Exporter
        print("exporting")
        guard let exporter = AVAssetExportSession(asset: mixComposition, presetName: AVAssetExportPresetHighestQuality) else { return }
        exporter.outputURL = url
        exporter.outputFileType = AVFileTypeQuickTimeMovie
        exporter.shouldOptimizeForNetworkUse = false
        exporter.videoComposition = mainComposition

        // 6 - Perform the Export
        controller.currentlyEditing = true
        exporter.exportAsynchronouslyWithCompletionHandler() {
            dispatch_async(dispatch_get_main_queue()) { _ in
                print("done")
                self.controller.currentlyEditing = false
                self.controller.merged = true
                self.button.blurView.superview?.hidden = true
                self.controller.player.replaceCurrentItemWithPlayerItem(AVPlayerItem(URL: url))
                self.controller.firstAsset = AVAsset(URL: url)
            }
        }
    }
}
func exportDidFinish(session: AVAssetExportSession) {
    if session.status == AVAssetExportSessionStatus.Failed {
        print(session.error)
    }
    if session.status == AVAssetExportSessionStatus.Completed {
        print("succed")
    }
}

Convert Image Here

class MyConverter: NSObject {

    var image:UIImage!

    convenience init(image:UIImage) {
        self.init()
        self.image = image
    }

    var outputURL: NSURL {
        let documentDirectory = NSSearchPathForDirectoriesInDomains(.DocumentDirectory, .UserDomainMask, true)[0]
        let savePath = (documentDirectory as NSString).stringByAppendingPathComponent("mergeVideo-pic.mov")
        return getURL(savePath)
    }

    func getURL(path:String) -> NSURL {
        let movieDestinationUrl = NSURL(fileURLWithPath: path)
        _ = try? NSFileManager().removeItemAtURL(movieDestinationUrl)
        let url = NSURL(fileURLWithPath: path)
        return url
    }

    func build(completion:() -> Void) {
        guard let videoWriter = try? AVAssetWriter(URL: outputURL, fileType: AVFileTypeQuickTimeMovie) else {
            fatalError("AVAssetWriter error")
        }
        let outputSettings = [AVVideoCodecKey : AVVideoCodecH264, AVVideoWidthKey : NSNumber(float: Float(image.size.width)), AVVideoHeightKey : NSNumber(float: Float(image.size.height))]

        guard videoWriter.canApplyOutputSettings(outputSettings, forMediaType: AVMediaTypeVideo) else {
            fatalError("Negative : Can't apply the Output settings...")
        }

        let videoWriterInput = AVAssetWriterInput(mediaType: AVMediaTypeVideo, outputSettings: outputSettings)
        let sourcePixelBufferAttributesDictionary = [kCVPixelBufferPixelFormatTypeKey as String : NSNumber(unsignedInt: kCVPixelFormatType_32ARGB), kCVPixelBufferWidthKey as String: NSNumber(float: Float(image.size.width)), kCVPixelBufferHeightKey as String: NSNumber(float: Float(image.size.height))]
        let pixelBufferAdaptor = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: videoWriterInput, sourcePixelBufferAttributes: sourcePixelBufferAttributesDictionary)

        if videoWriter.canAddInput(videoWriterInput) {
            videoWriter.addInput(videoWriterInput)
        }

        if videoWriter.startWriting() {
            videoWriter.startSessionAtSourceTime(kCMTimeZero)
            assert(pixelBufferAdaptor.pixelBufferPool != nil)
        }

        let media_queue = dispatch_queue_create("mediaInputQueue", nil)

        videoWriterInput.requestMediaDataWhenReadyOnQueue(media_queue, usingBlock: { () -> Void in
            var appendSucceeded = true
            //Time HERE IS ZERO, but in Merge file, it is 0.66667
            let presentationTime = CMTimeMake(0, 600)

            var pixelBuffer: CVPixelBuffer? = nil
            let status: CVReturn = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pixelBufferAdaptor.pixelBufferPool!, &pixelBuffer)

            if let pixelBuffer = pixelBuffer where status == 0 {
                let managedPixelBuffer = pixelBuffer
            CVPixelBufferLockBaseAddress(managedPixelBuffer, 0)

                let data = CVPixelBufferGetBaseAddress(managedPixelBuffer)
                let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
                let context = CGBitmapContextCreate(data, Int(self.image.size.width), Int(self.image.size.height), 8, CVPixelBufferGetBytesPerRow(managedPixelBuffer), rgbColorSpace, CGImageAlphaInfo.PremultipliedFirst.rawValue)

                CGContextClearRect(context, CGRectMake(0, 0, CGFloat(self.image.size.width), CGFloat(self.image.size.height)))


                CGContextDrawImage(context, CGRectMake(0, 0, self.image.size.width, self.image.size.height), self.image.CGImage)

                CVPixelBufferUnlockBaseAddress(managedPixelBuffer, 0)

                appendSucceeded =     pixelBufferAdaptor.appendPixelBuffer(pixelBuffer, withPresentationTime: presentationTime)
            } else {
                print("Failed to allocate pixel buffer")
                appendSucceeded = false
            }
            if !appendSucceeded {
                print("append failed")
            }
            videoWriterInput.markAsFinished()
            videoWriter.finishWritingWithCompletionHandler { () -> Void in
                print("FINISHED!!!!!")
                completion()
            }
        })
    }
}

Note: I have found out that if I do a print(presentationTime) INSIDE the ImageConverter it prints 0, and then print the time of the duration inside the merger, I get 0.666667

Note: No answers yet, but I will continuously be making this question a bounty until I find an answer or someone else helps me! Thanks!

Answer 1

Right, so I actually dealt with this problem a while ago. The issue is indeed with how you are creating the video from the picture. What you need to do is add the pixel buffer at time zero, and then AGAIN at the end, otherwise you will end up with an empty video until the very last frame, like you are experiencing.

The following code will be my best attempt to update your code. At the very end I will post my solution which is in Objective-C in case it helps anyone else.

func build(completion:() -> Void) {
    guard let videoWriter = try? AVAssetWriter(URL: outputURL, fileType: AVFileTypeQuickTimeMovie) else {
        fatalError("AVAssetWriter error")
    }

    // This might not be a problem for you but width HAS to be divisible by 16 or the movie will come out distorted... don't ask me why. So this is a safeguard
    let pixelsToRemove: Double = fmod(image.size.width, 16)
    let pixelsToAdd: Double = 16 - pixelsToRemove
    let size: CGSize = CGSizeMake(image.size.width + pixelsToAdd, image.size.height)

    let outputSettings = [AVVideoCodecKey : AVVideoCodecH264, AVVideoWidthKey : NSNumber(float: Float(size.width)), AVVideoHeightKey : NSNumber(float: Float(size.height))]

    guard videoWriter.canApplyOutputSettings(outputSettings, forMediaType: AVMediaTypeVideo) else {
        fatalError("Negative : Can't apply the Output settings...")
    }

    let videoWriterInput = AVAssetWriterInput(mediaType: AVMediaTypeVideo, outputSettings: outputSettings)
    let sourcePixelBufferAttributesDictionary = [kCVPixelBufferPixelFormatTypeKey as String : NSNumber(unsignedInt: kCVPixelFormatType_32ARGB), kCVPixelBufferWidthKey as String: NSNumber(float: Float(size.width)), kCVPixelBufferHeightKey as String: NSNumber(float: Float(size.height))]
    let pixelBufferAdaptor = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: videoWriterInput, sourcePixelBufferAttributes: sourcePixelBufferAttributesDictionary)

    if videoWriter.canAddInput(videoWriterInput) {
        videoWriter.addInput(videoWriterInput)
    }

    if videoWriter.startWriting() {
        videoWriter.startSessionAtSourceTime(kCMTimeZero)
        assert(pixelBufferAdaptor.pixelBufferPool != nil)
    }

    // For simplicity, I'm going to remove the media queue you created and instead explicitly wait until I can append since i am only writing one pixel buffer at two different times

    var pixelBufferCreated = true
    var pixelBuffer: CVPixelBuffer? = nil
    let status: CVReturn = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pixelBufferAdaptor.pixelBufferPool!, &pixelBuffer)

    if let pixelBuffer = pixelBuffer where status == 0 {
        let managedPixelBuffer = pixelBuffer
        CVPixelBufferLockBaseAddress(managedPixelBuffer, 0)

        let data = CVPixelBufferGetBaseAddress(managedPixelBuffer)
        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGBitmapContextCreate(data, Int(size.width), Int(size.height), 8, CVPixelBufferGetBytesPerRow(managedPixelBuffer), rgbColorSpace, CGImageAlphaInfo.PremultipliedFirst.rawValue)

        CGContextClearRect(context, CGRectMake(0, 0, CGFloat(size.width), CGFloat(size.height)))

        CGContextDrawImage(context, CGRectMake(0, 0, size.width, size.height), self.image.CGImage)

        CVPixelBufferUnlockBaseAddress(managedPixelBuffer, 0)
    } else {
        print("Failed to allocate pixel buffer")
        pixelBufferCreated = false
    }

    if (pixelBufferCreated) {
        // Here is where the magic happens, we have our pixelBuffer it's time to start writing

        // FIRST - add at time zero
        var appendSucceeded = pixelBufferAdaptor.appendPixelBuffer(pixelBuffer, withPresentationTime: kCMTimeZero];
        if (!appendSucceeded) {
            // something went wrong, up to you to handle. Should probably return so the rest of the code is not executed though
        }
        // SECOND - wait until the writer is ready for more data with an empty while
        while !writerInput.readyForMoreMediaData {} 

        // THIRD - make a CMTime with the desired length of your picture-video. I am going to arbitrarily make it 5 seconds here
        let frameTime: CMTime = CMTimeMake(5, 1) // 5 seconds

        // FOURTH - add the same exact pixel to the end of the video you are creating
        appendSucceeded = pixelBufferAdaptor.appendPixelBuffer(pixelBuffer, withPresentationTime: frameTime];
        if (!appendSucceeded) {
            // something went wrong, up to you to handle. Should probably return so the rest of the code is not executed though
        }

        videoWriterInput.markAsFinished() {
            videoWriter.endSessionAtSourceTime(frameTime)
        }
        videoWriter.finishWritingWithCompletionHandler { () -> Void in
            if videoWriter.status != .Completed {
                // Error writing the video... handle appropriately 
            } else {
                print("FINISHED!!!!!")
                completion()
            }
        }
    }
}

---

How I managed to do this in Obj-C

Note: I had to make some edits to make this standalone, so this method will return a string holding the path the video WILL BE written to. It is returned before the video writing finishes so it may be possible to access it before it is ready if you are not careful

-(NSString *)makeMovieFromImageData:(NSData *)imageData {
    NSError *error;
    UIImage *image = [UIImage imageWithData:imageData];


    // width has to be divisible by 16 or the movie comes out distorted... don't ask me why
    double pixelsToRemove = fmod(image.size.width, 16);

    double pixelsToAdd = 16 - pixelsToRemove; 

    CGSize size = CGSizeMake(image.size.width+pixelsToAdd, image.size.height);

    BOOL hasFoundValidPath = NO;
    NSURL *tempFileURL;
    NSString *outputFile;

    while (!hasFoundValidPath) {

        NSString *guid = [[NSUUID new] UUIDString];
        outputFile = [NSString stringWithFormat:@"picture_%@.mp4", guid];

        NSString *outputDirectory = [NSSearchPathForDirectoriesInDomains(NSTemporaryDirectory, NSUserDomainMask, YES) objectAtIndex:0];

        NSString *tempPath = [outputDirectory stringByAppendingPathComponent:outputFile];

        // Will fail if destination already has a file
        if ([[NSFileManager defaultManager] fileExistsAtPath:tempPath]) {
            continue;
        } else {
            hasFoundValidPath = YES;
        }
        tempFileURL = [NSURL fileURLWithPath:tempPath];
    }


    // Start writing
    AVAssetWriter *videoWriter = [[AVAssetWriter alloc] initWithURL:tempFileURL
                                                           fileType:AVFileTypeQuickTimeMovie
                                                              error:&error];

    if (error) {
       // handle error
    }

    NSDictionary *videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
                                   AVVideoCodecH264, AVVideoCodecKey,
                                   [NSNumber numberWithInt:size.width], AVVideoWidthKey,
                                   [NSNumber numberWithInt:size.height], AVVideoHeightKey,
                                   nil];

    AVAssetWriterInput* writerInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
                                                                         outputSettings:videoSettings];

    NSDictionary *bufferAttributes = [NSDictionary dictionaryWithObjectsAndKeys:
                                      [NSNumber numberWithInt:kCVPixelFormatType_32ARGB], kCVPixelBufferPixelFormatTypeKey, nil];

    AVAssetWriterInputPixelBufferAdaptor *adaptor = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:writerInput
                                                                                                                     sourcePixelBufferAttributes:bufferAttributes];
    if ([videoWriter canAddInput:writerInput]) {
        [videoWriter addInput:writerInput];
    } else {
        // handle error
    }

    [videoWriter startWriting];

    [videoWriter startSessionAtSourceTime:kCMTimeZero];

    CGImageRef img = [image CGImage];

    // Now I am going to create the bixelBuffer
    NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                            [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                            [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
                            nil];
    CVPixelBufferRef buffer = NULL;

    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width,
                                          size.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options,
                                          &pxbuffer);

    if ( !(status == kCVReturnSuccess && pxbuffer != NULL) ) {
        NSLog(@"There be some issue. We didn't get a buffer from the image");
    }


    CVPixelBufferLockBaseAddress(buffer, 0);
    void *pxdata = CVPixelBufferGetBaseAddress(buffer);

    CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();

    CGContextRef context = CGBitmapContextCreate(pxdata, size.width,
                                                 size.height, 8, 4*size.width, rgbColorSpace,
                                                 (CGBitmapInfo)kCGImageAlphaPremultipliedFirst);
    CGContextSetRGBFillColor(context, 0, 0, 0, 0);

    CGContextConcatCTM(context, CGAffineTransformIdentity);

    CGContextDrawImage(context, CGRectMake(0, 0, size.width,
                                           size.height), image);
    CGColorSpaceRelease(rgbColorSpace);
    CGContextRelease(context);

    CVPixelBufferUnlockBaseAddress(buffer, 0);

    // At this point we have our buffer so we are going to start by adding to time zero

    [adaptor appendPixelBuffer:buffer withPresentationTime:kCMTimeZero];

    while (!writerInput.readyForMoreMediaData) {} // wait until ready

    CMTime frameTime = CMTimeMake(5, 1); // 5 second frame

    [adaptor appendPixelBuffer:buffer withPresentationTime:frameTime];
    CFRelease(buffer);

    [writerInput markAsFinished];

    [videoWriter endSessionAtSourceTime:frameTime];

    [videoWriter finishWritingWithCompletionHandler:^{
        if (videoWriter.status != AVAssetWriterStatusCompleted) {
            // Error
        }
    }]; // end videoWriter finishWriting Block

    // NOTE: the URL is actually being returned before the videoWriter finishes writing so be careful to not access it until it's ready
    return outputFile;
}