Fastest way to calculate average RGB pixel value for AVCaptureVideoDataOutput feed - CPU/GPU

Question

I want the average pixel value for the entire image in the feed from AVCaptureVideoDataOutput, and I'm currently catching the image and looping through pixels to sum them.

I was wondering if there's a more efficient way to do this with the GPU/openGL, given that this is a parallelisable image processing task. (perhaps a heavy gaussian blur, and read the central pixel value?)

One specific requirement is for a high precision result, making use of the high level of averaging. Note the CGFloat result below.

Current swift 2 code:

Edit: Added an implementation with CIAreaAverage, as suggested below by Simon. It's separated by the useGPU bool.

func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!) {

    var redmean:CGFloat = 0.0;
    var greenmean:CGFloat = 0.0;
    var bluemean:CGFloat = 0.0;

    if (useGPU) {
            let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
            let cameraImage = CIImage(CVPixelBuffer: pixelBuffer!)
            let filter = CIFilter(name: "CIAreaAverage")
            filter!.setValue(cameraImage, forKey: kCIInputImageKey)
            let outputImage = filter!.valueForKey(kCIOutputImageKey) as! CIImage!

            let ctx = CIContext(options:nil)
            let cgImage = ctx.createCGImage(outputImage, fromRect:outputImage.extent)

            let rawData:NSData = CGDataProviderCopyData(CGImageGetDataProvider(cgImage))!
            let pixels = UnsafePointer<UInt8>(rawData.bytes)
            let bytes = UnsafeBufferPointer<UInt8>(start:pixels, count:rawData.length)
            var BGRA_index = 0
            for pixel in UnsafeBufferPointer(start: bytes.baseAddress, count: bytes.count) {
                switch BGRA_index {
                case 0:
                    bluemean = CGFloat (pixel)
                case 1:
                    greenmean = CGFloat (pixel)
                case 2:
                    redmean = CGFloat (pixel)
                case 3:
                    break
                default:
                    break
                }
                BGRA_index++

            }
     } else {
            let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
            CVPixelBufferLockBaseAddress(imageBuffer!, 0)

            let baseAddress = CVPixelBufferGetBaseAddressOfPlane(imageBuffer!, 0)
            let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer!)
            let width = CVPixelBufferGetWidth(imageBuffer!)
            let height = CVPixelBufferGetHeight(imageBuffer!)
            let colorSpace = CGColorSpaceCreateDeviceRGB()

            let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.PremultipliedFirst.rawValue).rawValue | CGBitmapInfo.ByteOrder32Little.rawValue

            let context = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, bitmapInfo)
            let imageRef = CGBitmapContextCreateImage(context)
            CVPixelBufferUnlockBaseAddress(imageBuffer!, 0)
            let data:NSData = CGDataProviderCopyData(CGImageGetDataProvider(imageRef))!
            let pixels = UnsafePointer<UInt8>(data.bytes)
            let bytes = UnsafeBufferPointer<UInt8>(start:pixels, count:data.length)
            var redsum:CGFloat = 0
            var greensum:CGFloat  = 0
            var bluesum:CGFloat  = 0
            var BGRA_index = 0
            for pixel in UnsafeBufferPointer(start: bytes.baseAddress, count: bytes.count) {
            switch BGRA_index {
            case 0:
                bluesum += CGFloat (pixel)
            case 1:
                greensum += CGFloat (pixel)
            case 2:
                redsum += CGFloat (pixel)
            case 3:
                //alphasum += UInt64(pixel)
                break
            default:
                break
            }

            BGRA_index += 1
            if BGRA_index == 4 { BGRA_index = 0 }
        }
        redmean = redsum / CGFloat(bytes.count)
        greenmean = greensum / CGFloat(bytes.count)
        bluemean = bluesum / CGFloat(bytes.count)            
        }

print("R:\(redmean) G:\(greenmean) B:\(bluemean)")

Answer 1

The issue and the reason for the poor performance of your CIAreaAverage filter is the missing definition of the input extent. As a consequence the output of the filter has the same size as the input image and therefore you loop over a full-blown image instead of a 1-by-1 pixel image. Therefore the execution takes the same amount of time as your initial version.

As described in the documentation of CIAreaAverage you can specify an inputExtent parameter. How this can done in swift can be found in this answer of a similar question:

    let cameraImage = CIImage(CVPixelBuffer: pixelBuffer!)
    let extent = cameraImage.extent
    let inputExtent = CIVector(x: extent.origin.x, y: extent.origin.y, z: extent.size.width, w: extent.size.height)
    let filter = CIFilter(name: "CIAreaAverage", withInputParameters: [kCIInputImageKey: cameraImage, kCIInputExtentKey: inputExtent])!
    let outputImage = filter.outputImage!

If you want to squeeze out even more performance, you can ensure that you reuse your CIContext, instead of recreating it for each captured frame.

Answer 2

There's a Core Image filter that does this very job, CIAreaAverage, which returns a single-pixel image that contains the average color for the region of interest (your region of interest will be the entire image).

FYI, I have a blog post that discusses applying Core Image filters to a live camera feed here. In a nutshell, the filter requires a CIImage which you can create inside captureImage based on sampleBuffer:

let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let cameraImage = CIImage(CVPixelBuffer: pixelBuffer!)

...and it's that cameraImage you'll need to pass to CIAreaAverage.

Cheers,

Simon