Fixing orientation when stitching (merging) videos using AVMutableComposition

Question

TLDR - SEE EDIT

I am creating a test app in Swift where I want to stitch multiple videos together from my apps documents directory using AVMutableComposition.

I have had success in doing this to some degree, all my videos are stitched together and everything is showing the correct size portrait and landscape.

My issue is, however, that all the videos are showing in the orientation of the last video in the compilation.

I know that to fix this I will need to add layer instructions for each track I add, however I can't seem to get this right, with the answers I have found the entire compilation seems to come out in a portrait orientation with landscape videos simply scaled to fit in the portrait view, so when I turn my phone on its side to view the landscape videos they are still small since they have been scaled to a portrait size.

This is not the outcome I am looking for, I want the expected functionality i.e. if a video is landscape it shows scaled when in portrait mode but if the phone is rotated I want that landscape video to fill the screen (as it would when simply viewing a landscape video in photos) and the same for portrait so that when viewing in portrait it is full screen and when turned sideways the video is scaled to landscape size (like it does when viewing a portrait video in photos).

In summary the desired outcome I want is that when viewing a compilation that has landscape and portrait videos I can view the entire compilation with my phone on its side and the landscape videos are full screen and portrait is scaled, or when viewing the same video in portrait the portrait videos are full screen and the landscape videos are scaled to size.

With all the answers I found this was not the case and they all seemed to have very unexpected behaviour when importing a video from photos to add to the compilation, and the same random behaviour when adding videos that were shot with the front facing camera (to be clear with my current implementation videos imported from library and "selfie" videos appear at the correct size without these problems).

I'm looking for a way to rotate/scale these videos so that they are always showing in the correct orientation and scale depending on which way round the user is holding their phone.

EDIT: I am now know that i can't have both landscape and portrait orientations in one single video, so the expected outcome I'm looking for would be to have the final video in landscape orientation. i have figured out how to make switch all the orientations and scales to get everything the same way up but my output is a portrait video if anyone could help me change this so my output is landscape it would be appreciated.

Below is my function to get the instruction for each video:

func videoTransformForTrack(asset: AVAsset) -> CGAffineTransform
{
    var return_value:CGAffineTransform?

    let assetTrack = asset.tracksWithMediaType(AVMediaTypeVideo)[0]

    let transform = assetTrack.preferredTransform
    let assetInfo = orientationFromTransform(transform)

    var scaleToFitRatio = UIScreen.mainScreen().bounds.width / assetTrack.naturalSize.width
    if assetInfo.isPortrait
    {
        scaleToFitRatio = UIScreen.mainScreen().bounds.width / assetTrack.naturalSize.height
        let scaleFactor = CGAffineTransformMakeScale(scaleToFitRatio, scaleToFitRatio)
        return_value = CGAffineTransformConcat(assetTrack.preferredTransform, scaleFactor)
    }
    else
    {
        let scaleFactor = CGAffineTransformMakeScale(scaleToFitRatio, scaleToFitRatio)
        var concat = CGAffineTransformConcat(CGAffineTransformConcat(assetTrack.preferredTransform, scaleFactor), CGAffineTransformMakeTranslation(0, UIScreen.mainScreen().bounds.width / 2))
        if assetInfo.orientation == .Down
        {
            let fixUpsideDown = CGAffineTransformMakeRotation(CGFloat(M_PI))
            let windowBounds = UIScreen.mainScreen().bounds
            let yFix = assetTrack.naturalSize.height + windowBounds.height
            let centerFix = CGAffineTransformMakeTranslation(assetTrack.naturalSize.width, yFix)
            concat = CGAffineTransformConcat(CGAffineTransformConcat(fixUpsideDown, centerFix), scaleFactor)
        }
        return_value = concat
    }
    return return_value!
}

And the exporter:

    // Create AVMutableComposition to contain all AVMutableComposition tracks
    let mix_composition = AVMutableComposition()
    var total_time = kCMTimeZero

    // Loop over videos and create tracks, keep incrementing total duration
    let video_track = mix_composition.addMutableTrackWithMediaType(AVMediaTypeVideo, preferredTrackID: CMPersistentTrackID())

    var instruction = AVMutableVideoCompositionLayerInstruction(assetTrack: video_track)
    for video in videos
    {
        let shortened_duration = CMTimeSubtract(video.duration, CMTimeMake(1,10));
        let videoAssetTrack = video.tracksWithMediaType(AVMediaTypeVideo)[0]

        do
        {
            try video_track.insertTimeRange(CMTimeRangeMake(kCMTimeZero, shortened_duration),
                ofTrack: videoAssetTrack ,
                atTime: total_time)

            video_track.preferredTransform = videoAssetTrack.preferredTransform

        }
        catch _
        {
        }

        instruction.setTransform(videoTransformForTrack(video), atTime: total_time)

        // Add video duration to total time
        total_time = CMTimeAdd(total_time, shortened_duration)
    }

    // Create main instrcution for video composition
    let main_instruction = AVMutableVideoCompositionInstruction()
    main_instruction.timeRange = CMTimeRangeMake(kCMTimeZero, total_time)
    main_instruction.layerInstructions = [instruction]
    main_composition.instructions = [main_instruction]
    main_composition.frameDuration = CMTimeMake(1, 30)
    main_composition.renderSize = CGSize(width: UIScreen.mainScreen().bounds.width, height: UIScreen.mainScreen().bounds.height)

    let exporter = AVAssetExportSession(asset: mix_composition, presetName: AVAssetExportPreset640x480)
    exporter!.outputURL = final_url
    exporter!.outputFileType = AVFileTypeMPEG4
    exporter!.shouldOptimizeForNetworkUse = true
    exporter!.videoComposition = main_composition

    // 6 - Perform the Export
    exporter!.exportAsynchronouslyWithCompletionHandler()
    {
        // Assign return values based on success of export
        dispatch_async(dispatch_get_main_queue(), { () -> Void in
                self.exportDidFinish(exporter!)
        })
    }

Sorry for the long explanation I just wanted to make sure I was very clear with what I was asking because other answers have not worked for me.

Answer 1

Im not sure your orientationFromTransform() give you the correct orientation.

I think you try to modify it or try anything like:

extension AVAsset {

    func videoOrientation() -> (orientation: UIInterfaceOrientation, device: AVCaptureDevicePosition) {
        var orientation: UIInterfaceOrientation = .Unknown
        var device: AVCaptureDevicePosition = .Unspecified

        let tracks :[AVAssetTrack] = self.tracksWithMediaType(AVMediaTypeVideo)
        if let videoTrack = tracks.first {

            let t = videoTrack.preferredTransform

            if (t.a == 0 && t.b == 1.0 && t.d == 0) {
                orientation = .Portrait

                if t.c == 1.0 {
                    device = .Front
                } else if t.c == -1.0 {
                    device = .Back
                }
            }
            else if (t.a == 0 && t.b == -1.0 && t.d == 0) {
                orientation = .PortraitUpsideDown

                if t.c == -1.0 {
                    device = .Front
                } else if t.c == 1.0 {
                    device = .Back
                }
            }
            else if (t.a == 1.0 && t.b == 0 && t.c == 0) {
                orientation = .LandscapeRight

                if t.d == -1.0 {
                    device = .Front
                } else if t.d == 1.0 {
                    device = .Back
                }
            }
            else if (t.a == -1.0 && t.b == 0 && t.c == 0) {
                orientation = .LandscapeLeft

                if t.d == 1.0 {
                    device = .Front
                } else if t.d == -1.0 {
                    device = .Back
                }
            }
        }

        return (orientation, device)
    }
}