How to composite videos using multiple AVVideoCompositions

Question

I'm trying to figure out how to composite multiple videos (AVAssets) into a single video such that each of the videos goes through its own video composition. However, I can't see a way to accomplish this and was wondering if anyone had any ideas.

Consider the following:

The above picture illustrates what I'm trying to do. I want to take the video track from four different videos and merge them into a single video such that they play in a grid-like layout.

Right now, I'm able to achieve this by doing the following

• Create a single AVMutableComposition

• Add four video tracks to the mutable composition

• Create four AVMutableVideoCompositionLayerInstruction instances with the appropriate transform applied to scale and translate the track.

• Create an AVMutableVideoComposition, add the layer instructions to it and then set that video composition on the appropriate AVAssetExportSession

While this works, it doesn't offer me the ability to do any custom effects on each video track before compositing it into the output clip. For example, in the picture above I'd like to add a unique border color around each track. With a single video (and thus no layer instructions), then I can just add the border using something like AVVideoComposition applyingCIFiltersWithHandler, which works great. But I can't use that because I need the video composition for setting the transform and scale instructions.

(applyingCIFiltersWithHandler doesn't appear to offer a way to get access to the tracks that make up the frame being rendered.)

What I think I want is the ability to do some type of custom rendering within a AVMutableVideoCompositionLayerInstruction, but the only options currently available are transform and opacity ones.

(Note that any solution requires that I have access to the current composition time during rendering for each video track because some of the "effects" I'd like to render are a frame counter and a timecode display.)

Any help, ideas or tips would be appreciated.

Answer 1

You need to implement a custom compositor that implements the AVVideoCompositing protocol. https://developer.apple.com/reference/avfoundation/avvideocompositing

The startRequest function within that protocol receives an https://developer.apple.com/reference/avfoundation/avasynchronousvideocompositionrequest and this allows you to access the information from each track https://developer.apple.com/reference/avfoundation/avasynchronousvideocompositionrequest/1390379-sourceframe, then combine them with your own custom instructions and composite (put them together) in your own way, returning a final single buffer that has the frame composited from the tracks in the way you like.