Low FPS when accesing iPhone video output image buffer

Question

I'm trying to do some image processing on iPhone. I'm using http://developer.apple.com/library/ios/#qa/qa2010/qa1702.html to capture the camera frames.

My problem is that when I'm trying to access the captured buffer, the camera FPS drops from 30 to about 20. Does anybody knows how I can fix it?

I use the lowest capture quality I could find (AVCaptureSessionPresetLow = 192x144) in kCVPixelFormatType_32BGRA format. If anybody knows a lower quality I could use, I'm willing to try it.

When I do the same image access on other platforms, like Symbian, it works OK.

Here is my code:

#pragma mark -
#pragma mark AVCaptureSession delegate
- (void)captureOutput:(AVCaptureOutput *)captureOutput 
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer 
    fromConnection:(AVCaptureConnection *)connection 
{ 
 /*We create an autorelease pool because as we are not in the main_queue our code is
  not executed in the main thread. So we have to create an autorelease pool for the thread we are in*/
 NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
 //Lock the image buffer
    if (CVPixelBufferLockBaseAddress(imageBuffer, 0) == kCVReturnSuccess)
    {  

  // calculate FPS and display it using main thread
  [self performSelectorOnMainThread:@selector(updateFps:) withObject: (id) nil waitUntilDone:NO];


  UInt8 *base = (UInt8 *)CVPixelBufferGetBaseAddress(imageBuffer); //image buffer start address

  size_t width  = CVPixelBufferGetWidth(imageBuffer); 
  size_t height = CVPixelBufferGetHeight(imageBuffer);

  int size = (height*width);
  UInt8* pRGBtmp = m_pRGBimage;

        /*
        Here is the problem; m_pRGBimage is RGB image I want to process.
        In the 'for' loop I convert the image from BGRA to RGB. As a resault, the FPS drops to 20.
        */ 
  for (int i=0;i<size;i++)
  {   
    pRGBtmp[0] = base[2];
    pRGBtmp[1] = base[1];
    pRGBtmp[2] = base[0];
    base = base+4;
    pRGBtmp = pRGBtmp+3;     
  }


  // Display received action
  [self performSelectorOnMainThread:@selector(displayAction:) withObject: (id) nil waitUntilDone:NO];
  //[self displayAction:&eyePlayOutput];
  //saveFrame( imageBuffer );

  //unlock the  image buffer
  CVPixelBufferUnlockBaseAddress(imageBuffer,0);

 }


 [pool drain];
} 

As a follow-on to the answers, I need to process the image in realtime, it is being displayed.

I noticed that when I use AVCaptureSessionPresetHigh, the most simple thing I do, like:

 for (int i=0;i<size;i++)
    x = base[0];

causes the framerate to drop to 4-5 FPS. I guess its because an image in that size is not cached.

Basically I need 96x48 image. is there a simple way to downscale the camera output image, a way that uses hardware acceleration, so I could work with the small one?

Answer 1

Anything that iterates over every pixel in an image will be fairly slow on all but the fastest iOS devices. For example, I benchmarked iterating over every pixel in a 640 x 480 video frame (307,200 pixels) with a simple per-pixel color test and found that this only runs at around 4 FPS on an iPhone 4.

You're looking at processing 27,648 pixels in your case, which should run fast enough to hit 30 FPS on an iPhone 4, but that's a much faster processor than what was in the original iPhone and iPhone 3G. The iPhone 3G will probably still struggle with this processing load. You also don't say how fast the processor was in your Symbian devices.

I'd suggest reworking your processing algorithm to avoid the colorspace conversion. There should be no need to reorder the color components in order to process them.

Additionally, you could selectively process only a few pixels by sampling at certain intervals within the rows and columns of the image.

Finally, if you are targeting the newer iOS devices that have support for OpenGL ES 2.0 (iPhone 3G S and newer), you might want to look at using a GLSL fragment shader to process the video frame entirely on the GPU. I describe the process here, along with sample code for realtime color-based object tracking. The GPU can handle this kind of processing 14 - 28 times faster than the CPU, in my benchmarks.

Answer 2

disclaimer: THIS ANSWER IS A GUESS :)

You're doing quite a lot of work while the buffer is locked; is this holding up the thread that is capturing the image from the camera?

You could you copy the data out of the buffer while you work on it so you can unlock it asap i.e. something like

if (CVPixelBufferLockBaseAddress(imageBuffer, 0) == kCVReturnSuccess) {
    // Get the base address and size of the buffer
    UInt8 *buffer_base = (UInt8 *)CVPixelBufferGetBaseAddress(imageBuffer); //image buffer start address
    size_t width  = CVPixelBufferGetWidth(imageBuffer); 
    size_t height = CVPixelBufferGetHeight(imageBuffer);

    // Copy it's contents out
    Uint8 *base = malloc(width * height * 4);
    memcpy(base, buffer_base, size);

    // Unlock the buffer
    CVPixelBufferUnlockBaseAddress(imageBuffer,0);

    // base now points to a copy of the buffers' data - do what you want to it . . .
    ...

    // remember to free base once you're done ;)
    free(base);

If it's the lock that's holding up the capture then this should help.

NB You could speed this up if you know all the buffers will be the same size you can just call malloc once to get the memory and then just reuse it each time and only free it when you have finished processing all the buffers.

---

Or if that's not the problem you could try lowering the priority of this thread

[NSThread setThreadPriority:0.25];