GPUImage Video with transparency over UIView

Question

I am working on an iOS project that uses AV-Out to show contents in a 1280x720 window on a second screen.

I have a MPMoviePlayerController's view as background and on top of that different other elements like UIImages and UILabels. The background movie plays in a loop.

Now I want to overlay the whole view including all visible elements with another fullscreen animation that has transparency so that only parts of the underlying view are visible.

I first tried a png animation with UIImageView. I was surprised to find that actually works on iPhone5, but of course the pngs are so big in size that this uses way too much ram and it crashes on everything below iPhone4s. So i need another way.

I figured out how to play a second movie at the same time using AVFoundation. So far, so good. Now i can play the overlay video, but of course it is not trasparent yet.

I also learned that with the GPUImage library I can use GPUImageChromaKeyBlendFilter to filter a color out of a video to make it transparent and then combine it with another video.

What i don't understand yet is the best way to implement it in my case to get the result that i want.

Can i use the whole view hierarchy below the top video as first input for the GPUImageChromaKeyBlendFilter and a greenscreen-style video as second input and show the result live in 720p? how would i do that?

Or would it be better to use GPUImageChromaKeyFilter and just filter the greenscreen-style video, and play it in a view above all other views? Would the background of this video be transparent then?

Thanks for your help!

Answer 1

You'll need to build a custom player using AVFoundation.framework and then use a video with alpha channel. The AVFoundation framework allows much more robust handeling of video without many of the limitations of MPMedia framework. Building a custom player isn't as hard as people make it out to be. I've written a tutorial on it here:http://www.sdkboy.com/?p=66

OTHER WAY..........

The SimpleVideoFileFilter example in the framework shows how to load a movie, filter it, and encode it back to disk. Modifying this to perform chroma keying gives the following:

NSURL *sampleURL = [[NSBundle mainBundle] URLForResource:@"sample" withExtension:@"m4v"];
movieFile = [[GPUImageMovie alloc] initWithURL:sampleURL];

filter = [[GPUImageChromaKeyBlendFilter alloc] init];
[filter setColorToReplaceRed:0.0 green:0.0 blue:1.0];
[filter setThresholdSensitivity:0.4];

[movieFile addTarget:filter];

UIImage *inputImage = [UIImage imageNamed:@"background.jpg"];
sourcePicture = [[GPUImagePicture alloc] initWithImage:inputImage smoothlyScaleOutput:YES];
[sourcePicture addTarget:filter];

NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:@"Documents/Movie.m4v"];
NSURL *movieURL = [NSURL fileURLWithPath:pathToMovie];
movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:movieURL size:CGSizeMake(480.0, 640.0)];
[filter addTarget:movieWriter];

[movieWriter startRecording];
[movieFile startProcessing];

[movieWriter setCompletionBlock:^{
    [filter removeTarget:movieWriter];
    [movieWriter finishRecording];
}];

The above code will load a movie from the application's resources called sample.m4v, feed it into a chroma key filter that is set to key off of pure blue with a sensitivity of 0.4, attach a background image to use for the chroma keying, and then send all that to a movie encoder which writes Movie.m4v in the application's /Documents directory.

You can adjust the threshold and specific blue tint to match your needs, as well as replace the input image with another movie or other source as needed. This process can also be applied to live video from the iOS device's camera, and you can display the results to the screen if you'd like.

On an iPhone 4 running iOS 5.0, chroma keying takes 1.8 ms (500+ FPS) for 640x480 frames of video, 65 ms for 720p frames (15 FPS). The newer A5-based devices are 6-10X faster than that for these operations, so they can handle 1080p video without breaking a sweat. I use iOS 5.0's fast texture caches for both frame uploads and retrievals, which accelerates the processing on that OS version over 4.x.

The one caution I have about this is that I haven't quite gotten the audio to record right in my movie encoding, but I'm working on that right now.