How-to convert an iOS camera image to greyscale using the Accelerate Framework?

Question

It seems like this should be simpler than I'm finding it to be.

I have an AVFoundation frame coming back in the standard delegate method:

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection

where I would like to convert the frame to greyscale using the Accelerate.Framework.

There is a family of conversion methods in the framework, including vImageConvert_RGBA8888toPlanar8(), which looks like it might be what I would like to see, however, I can't find any examples of how to use them!

So far, I have the code:

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection
{

      @autoreleasepool {
            CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
            /*Lock the image buffer*/
            CVPixelBufferLockBaseAddress(imageBuffer,0);
            /*Get information about the image*/
            uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
            size_t width = CVPixelBufferGetWidth(imageBuffer);
            size_t height = CVPixelBufferGetHeight(imageBuffer);
            size_t stride = CVPixelBufferGetBytesPerRow(imageBuffer);

            // vImage In
            Pixel_8 *bitmap = (Pixel_8 *)malloc(width * height * sizeof(Pixel_8));
            const vImage_Buffer inImage = { bitmap, height, width, stride };

            //How can I take this inImage and convert it to greyscale?????
            //vImageConvert_RGBA8888toPlanar8()??? Is the correct starting format here??
      }    
}

So I have two questions: (1) In the code above, is RBGA8888 the correct starting format? (2) How can I actually make the Accelerate.Framework call to convert to greyscale?

Answer 1

There is an easier option here. If you change the camera acquire format to YUV, then you already have a greyscale frame that you can use as you like. When setting up your data output, use something like:

dataOutput.videoSettings = @{ (id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) };

You can then access the Y plane in your capture callback using:

CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
uint8_t *yPlane = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);

... do stuff with your greyscale camera image ...

CVPixelBufferUnlockBaseAddress(pixelBuffer);

Answer 2

The vImage method is to use vImageMatrixMultiply_Planar8 and a 1x3 matrix. vImageConvert_RGBA8888toPlanar8 is the function you use to convert a RGBA8888 buffer into 4 planar buffers. These are used by vImageMatrixMultiply_Planar8. vImageMatrixMultiply_ARGB8888 will do it too in one pass, but your gray channel will be interleaved with three other channels in the result. vImageConvert_RGBA8888toPlanar8 itself doesn't do any math. All it does is separate your interleaved image into separate image planes.

If you need to adjust the gamma as well, then probably vImageConvert_AnyToAny() is the easy choice. It will do the fully color managed conversion from your RGB format to a grayscale colorspace. See vImage_Utilities.h.

I like Tarks answer better though. It just leaves you in a position of having to color manage the Luminance manually (if you care).