How can I boost up OpenCV performance on iPhone?

Question

I'm using AVFoundation to capture video along with OpenCV to do image processing on iPhone. However, I'm considering using OpenGL shaders to do calculations (convolving) on 192x144 grayscale image.

My question is: does it make sense to use OpenGL, I mean: isn't rendering into IplImage (is it even possible?) too slow? Or, is there a way which will speed-up the convolving?

EDIT

In OpenCV, I'm using

IplConvKernel* kernel = cvCreateStructuringElementEx(3, 3, 1, 1, CV_SHAPE_CROSS, NULL);

with cvMorphologyEx.

Answer 1

Even with the use of NEON vector operations via something like the Accelerate framework (which OpenCV currently doesn't use, if I'm not missing something), it's going to be hard to beat the performance of shaders when running simple 3x3 convolution kernels. For example, I can run a Sobel edge detection kernel on a 640x480 frame of video in 2.5 ms on an iPhone 4 using shaders, which is more than fast enough for realtime image processing. Also, OpenGL ES shaders have significant advantages when processing for display, because you can keep everything on the GPU side the whole way through and avoid expensive data transfers for the drawing events.

If you want a simple way to so this, my open source GPUImage framework has some very fast built-in convolutions, like Sobel edge detection or an image sharpening kernel, and it lets you create your own 3x3 convolution kernels pretty easily. It wraps all of the OpenGL ES for you, so you don't need to know anything about that (unless you want to write your own custom effects, but even there you just need to know a little GLSL).

For example, to do only the X component of the Sobel edge detection kernel, you can set up your convolution filter with the following code:

GPUImage3x3ConvolutionFilter *filter = [[GPUImage3x3ConvolutionFilter alloc] init];
[filter setConvolutionKernel:(GPUMatrix3x3){
    {-1.0f,  0.0f, 1.0f},
    {-2.0f,  0.0f, 2.0f},
    {-1.0f,  0.0f, 1.0f}
 }];

You then just have to attach this to a camera, image, or movie input and display, raw data, or movie recorder output and the framework will handle the rest. For best performance, you might choose to write your own custom implementation that is optimized to the particular kernel you want to run, like I've done for the GPUImageSobelEdgeDetectionFilter.

Answer 2

I wasn't able to find any relevant material on this, so I guess it's impossible to say for sure if a implementation using shaders of this algorithm would have better performance with small images like 192x144.

Since this is not a complex implementation, if you have the time I encourage you to do it and measure the performance gain of both implementations.

Please report back to us with your findings if you decide to follow this path.