How to use OpenGL fragment shader to convert RGB to YUV420

Question

I know how to code a shader that converts planar data (YUV420 image) to packed one (RGB image) using fragment shader, yet how do I code the RGB to YUV420. I do not need the YUV420 image for rendering but to pass it to the h264 compression code.

Answer 1

From http://www.fourcc.org/fccyvrgb.php:

RGB to YUV Conversion

Y = (0.257 * R) + (0.504 * G) + (0.098 * B) + 16

Cr = V = (0.439 * R) - (0.368 * G) - (0.071 * B) + 128

Cb = U = -(0.148 * R) - (0.291 * G) + (0.439 * B) + 128

In a OpenGL shader those constant offsets at the end need to be multiplied by 1/256. Also if you look closely this is a vector-matrix multiplication. You can express this by

mat4 RGBtoYUV(0.257,  0.439, -0.148, 0.0,
              0.504, -0.368, -0.291, 0.0,
              0.098, -0.071,  0.439, 0.0,
              0.0625, 0.500,  0.500, 1.0 );

YUV = RGBtoYUV * RGB;

Some other formulas are given at http://en.wikipedia.org/wiki/YUV

Answer 2

Since the YUV pixel is not packed, you shall attach multiple color buffers to the shader program (i.e. One for luminance and two for chrominance).

The sizes of the chrominance output buffers differs fromt the luminance one, due the chrominance sampling. Because this, you shall test for ARB_framebuffer_object extension. If not available you shall compute unsampled chrominance values and sample them after rendering. Note also that sampling in shader is not easy, since you have to know the source pixel position, indeed be carefull about texture coordinates.

So, create a framebuffer object, attach 3 renderbuffer objects (or textures), and declare fragment shader output variable as an array of length 3 of vec4.

Attached buffers can have a RED internal format, since they storeonly one component.

Once rendered, fetch renderbuffer data (or texture data), interleave YUV layer as needed (maybe using SSE), and you have done your job.

Maybe you want to generate unsampled chrominance values. In this case you shall sample chrominance buffers by yourself, maybe blending contiguos chrominance values.