iOS 5 + GLKView: How to access pixel RGB data for colour-based vertex picking?

Question

I've been converting my own personal OGLES 2.0 framework to take advantage of the functionality added by the new iOS 5 framework GLKit.

After pleasing results, I now wish to implement the colour-based picking mechanism described here. For this, you must access the back buffer to retrieve a touched pixel RGBA value, which is then used as a unique identifier for a vertex/primitive/display object. Of course, this requires temporary unique coloring of all vertices/primitives/display objects.

I have two questions, and I'd be very grateful for assistance with either:

1. I have access to a GLKViewController, GLKView, CAEAGLLayer (of the GLKView) and an EAGLContext. I also have access to all OGLES 2.0 buffer related commands. How do I combine these to identify the color of a pixel in the EAGLContext I'm tapping on-screen?

2. Given that I'm using Vertex Buffer Objects to do my rendering, is there a neat way to override the colour provided to my vertex shader which firstly doesn't involve modifying buffered vertex (colour) attributes, and secondly doesn't involve the addition of an IF statement into the vertex shader?

I assume the answer to (2) is "no", but for reasons of performance and non-arduous code revamping I thought it wise to check with someone more experienced.

Any suggestions would be gratefully received. Thank you for your time

UPDATE

Well I now know how to read pixel data from the active frame buffer using glReadPixels. So I guess I just have to do the special "unique colours" render to the back buffer, briefly switch to it and read pixels, then switch back. This will inevitably create a visual flicker, but I guess it's the easiest way; certainly quicker (and more sensible) than creating a CGImageContextRef from a screen snapshot and analyzing that way.

Still, any tips as regards the back buffer would be much appreciated.

Answer 1

Well, I've worked out exactly how to do this as concisely as possible. Below I explain how to achieve this and list all the code required :)

In order to allow touch interaction to select a pixel, first add a UITapGestureRecognizer to your GLKViewController subclass (assuming you want tap-to-select-pixel), with the following target method inside that class. You must make your GLKViewController subclass a UIGestureRecognizerDelegate:

@interface GLViewController : GLKViewController <GLKViewDelegate, UIGestureRecognizerDelegate>

After instantiating your gesture recognizer, add it to the view property (which in GLKViewController is actually a GLKView):

// Inside GLKViewController subclass init/awakeFromNib:
[[self view] addGestureRecognizer:[self tapRecognizer]];
[[self tapRecognizer] setDelegate:self];

Set the target action for your gesture recognizer; you can do this when creating it using a particular init... however I created mine using Storyboard (aka "the new Interface Builder in Xcode 4.2") and wired it up that way.

Anyway, here's my target action for the tap gesture recognizer:

-(IBAction)onTapGesture:(UIGestureRecognizer*)recognizer {
    const CGPoint loc = [recognizer locationInView:[self view]];
    [self pickAtX:loc.x Y:loc.y];
}

The pick method called in there is one I've defined inside my GLKViewController subclass:

-(void)pickAtX:(GLuint)x Y:(GLuint)y {
    GLKView *glkView = (GLKView*)[self view];
    UIImage *snapshot = [glkView snapshot];
    [snapshot pickPixelAtX:x Y:y];
}

This takes advantage of a handy new method snapshot that Apple kindly included in GLKView to produce a UIImage from the underlying EAGLContext.

What's important to note is a comment in the snapshot API documentation, which states:

This method should be called whenever your application explicitly needs the contents of the view; never attempt to directly read the contents of the underlying framebuffer using OpenGL ES functions.

This gave me a clue as to why my earlier attempts to invoke glReadPixels in attempts to access pixel data generated an EXC_BAD_ACCESS, and the indicator that sent me down the right path instead.

You'll notice in my pickAtX:Y: method defined a moment ago I call a pickPixelAtX:Y: on the UIImage. This is a method I added to UIImage in a custom category:

@interface UIImage (NDBExtensions)
-(void)pickPixelAtX:(NSUInteger)x Y:(NSUInteger)y;
@end

Here is the implementation; it's the final code listing required. The code came from this question and has been amended according to the answer received there:

@implementation UIImage (NDBExtensions)

- (void)pickPixelAtX:(NSUInteger)x Y:(NSUInteger)y {

    CGImageRef cgImage = [self CGImage];
    size_t width = CGImageGetWidth(cgImage);
    size_t height = CGImageGetHeight(cgImage);

    if ((x < width) && (y < height))
    {
        CGDataProviderRef provider = CGImageGetDataProvider(cgImage);
        CFDataRef bitmapData = CGDataProviderCopyData(provider);
        const UInt8* data = CFDataGetBytePtr(bitmapData);
        size_t offset = ((width * y) + x) * 4;
        UInt8 b = data[offset+0];
        UInt8 g = data[offset+1];
        UInt8 r = data[offset+2];
        UInt8 a = data[offset+3];
        CFRelease(bitmapData);
        NSLog(@"R:%i G:%i B:%i A:%i",r,g,b,a);
    }
}

@end

I originally tried some related code found in an Apple API doc entitled: "Getting the pixel data from a CGImage context" which required 2 method definitions instead of this 1, but much more code is required and there is data of type void * for which I was unable to implement the correct interpretation.

That's it! Add this code to your project, then upon tapping a pixel it will output it in the form:

R:24 G:46 B:244 A:255

Of course, you should write some means of extracting those RGBA int values (which will be in the range 0 - 255) and using them however you want. One approach is to return a UIColor from the above method, instantiated like so:

UIColor *color = [UIColor colorWithRed:red/255.0f green:green/255.0f blue:blue/255.0f alpha:alpha/255.0f];