Android Camera Preview YUV format into RGB on the GPU

Question

I have copy pasted some code I found on stackoverflow to convert the default camera preview YUV into RGB format and then uploaded it to OpenGL for processing. That worked fine, the issue is that most of the CPU was busy at converting the YUV images into the RGB format and it turned into the bottle neck.

I want to upload the YUV image into the GPU and then convert it into RGB in a fragment shader. I took the same Java YUV to RGB function I found which worked on the CPU and tried to make it work on the GPU.

It turned to be quite a little nightmare, since there are several differences on doing calculations on Java and the GPU. First, the preview image comes in byte[] in Java, but bytes are signed, so there might be negative values.

In addition, the fragment shader normally deals with [0..1] floating values for instead of a byte.

I am sure this is solveable and I almost solved it. But I spent a few hours trying to figure out what I was doing wrong and couldn't make it work.

Bottom line, I ask for someone to just write this shader function and preferably test it. For me it would be a tedious monkey job since I don't really understand why this conversion works the way it is, and I just try to mimic the same function on the GPU.

This is a very similar function to what I used on Java: Displaying YUV Image in Android

What I did some of the job on the CPU, such as turnning the 1.5*wh bytes YUV format into a wh*YUV, as follows:

static public void decodeYUV420SP(int[] rgba, byte[] yuv420sp, int width,
        int height) {
    final int frameSize = width * height;

    for (int j = 0, yp = 0; j < height; j++) {
        int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
        for (int i = 0; i < width; i++, yp++) {
            int y = (int) yuv420sp[yp]+127;
            if ((i & 1) == 0) {
                v = (int)yuv420sp[uvp++]+127;
                u = (int)yuv420sp[uvp++]+127;
            }
            rgba[yp] = 0xFF000000+(y<<16) | (u<<8) | v;
        }
    }
}

I added 127 because byte is signed. I then loaded the rgba into a OpenGL texture and tried to do the rest of the calculation on the GPU.

Any help would be appreaciated...

Answer 1

I used this code from wikipedia to calculate the conversion from YUV to RGB on the GPU:

private static int convertYUVtoRGB(int y, int u, int v) {
    int r,g,b;

    r = y + (int)1.402f*v;
    g = y - (int)(0.344f*u +0.714f*v);
    b = y + (int)1.772f*u;
    r = r>255? 255 : r<0 ? 0 : r;
    g = g>255? 255 : g<0 ? 0 : g;
    b = b>255? 255 : b<0 ? 0 : b;
    return 0xff000000 | (b<<16) | (g<<8) | r;
}

I converted the floats to 0.0..255.0 and then use the above code. The part on the CPU was to rearrange the original YUV pixels into a YUV matrix(also shown in wikipdia). Basically I used the wikipedia code and did the simplest float<->byte conersions to make it work out. Small mistakes like adding 16 to Y or not adding 128 to U and V would give undesirable results. So you need to take care of it. But it wasn't a lot of work once I used the wikipedia code as the base.

Answer 2

Converting on CPU sounds easy but I believe question is how to do it on GPU?

I did it recently in my project where I needed to get very fast QR code detection even when camera angle is 45 degrees to surface where code is printed, and it worked with great performance:

(following code is trimmed just to contain key lines, it is assumed that you have both Java and OpenGLES solid understanding)

1. Create a GL texture that will contain stored Camera image:

   int[] txt = new int[1];
   GLES20.glGenTextures(1,txt,0);
   GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,txt[0]);
   GLES20.glTextParameterf(... set min filter to GL_LINEAR );
   GLES20.glTextParameterf(... set mag filter to GL_LINEAR );
   GLES20.glTextParameteri(... set wrap_s to GL_CLAMP_TO_EDGE );
   GLES20.glTextParameteri(... set wrap_t to GL_CLAMP_TO_EDGE );
   

Pay attention that texture type is not GL_TEXTURE_2D. This is important, since only a GL_TEXTURE_EXTERNAL_OES type is supported by SurfaceTexture object, which will be used in the next step.

2. Setup SurfaceTexture:

   SurfaceTexture surfTex = new SurfaceTeture(txt[0]);
   surfTex.setOnFrameAvailableListener(this); 
   

Above assumes that 'this' is an object that implements 'onFrameAvailable' function.

    public void onFrameAvailable(SurfaceTexture st)
    {
            surfTexNeedUpdate = true;
            // this flag will be read in GL render pipeline
    }

3. Setup camera:

   Camera cam = Camera.open();
   cam.setPreviewTexture(surfTex);
   

This Camera API is deprecated if you target Android 5.0, so if you are, you have to use new CameraDevice API.

4. In your render pipeline, have following block to check if camera has frame available, and update surface texture with it. When surface texture is updated, will fill in GL texture that is linked with it.

   if( surfTexNeedUpdate )
   {
           surfTex.updateTexImage();
           surfTexNeedUpdate = false;
   }
   

5. To bind GL texture which has Camera -> SurfaceTeture link to, just do this in rendering pipe:

   GLES20.glBindTexture(GLES20.GL_TEXTURE_EXTERNAL_OS, txt[0]);
   

Goes without saying, you need to set current active texture.

6. In your GL shader program which will use above texture in it's fragment part, you must have first line:

   #extension GL_OES_EGL_imiage_external : require
   

Above is a must-have.

Texture uniform must be samplerExternalOES type:

    uniform samplerExternalOES u_Texture0;

Reading pixel from it is just like from GL_TEXTURE_2D type, and UV coordinates are in same range (from 0.0 to 1.0):

    vec4 px = texture2D(u_Texture0, v_UV);

7. Once you have your render pipeline ready to render a quad with above texture and shader, just start the camera:

   cam.startPreview();
   

8. You should see quad on your GL screen with live camera feed. Now you just need to grab the image with glReadPixels:

   GLES20.glReadPixels(0,0,width,height,GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, bytes);
   

Above line assumes that your FBO is RGBA, and that bytes is already initialized byte[] array to proper size, and that width and height are size of your FBO.

And voila! You have captured RGBA pixels from camera instead of converting YUV bytes received in onPreviewFrame callback...

You can also use RGB framebuffer object and avoid alpha if you don't need it.

It is important to note that camera will call onFrameAvailable in it's own thread which is not your GL render pipeline thread, thus you should not perform any GL calls in that function.