OpenGL ES 2.0 shader examples for image processing?

Question

I am learning shader programming and looking for examples, specifically for image processing. I'd like to apply some Photoshop effect to my photos, e.g. Curves, Levels, Hue/Saturation adjustments, etc.

Answer 1

I'll assume you have a simple uncontroversial vertex shader, as it's not really relevant to the question, such as:

void main()
{
    gl_Position = modelviewProjectionMatrix * position;
    texCoordVarying = vec2(textureMatrix * vec4(texCoord0, 0.0, 1.0));
}

So that does much the same as ES 1.x would if lighting was disabled, including the texture matrix that hardly anyone ever uses.

I'm not a Photoshop expert, so please forgive my statements of what I think the various tools do — especially if I'm wrong.

I think I'm right to say that the levels tool effectively stretches (and clips) the brightness histogram? In that case an example shader could be:

varying mediump vec2 texCoordVarying;
uniform sampler2D tex2D;

const mediump mat4 rgbToYuv = mat4( 0.257,  0.439,  -0.148, 0.06, 
                                    0.504, -0.368,  -0.291, 0.5,
                                    0.098, -0.071,   0.439, 0.5, 
                                     0.0,     0.0,     0.0, 1.0);

const mediump mat4 yuvToRgb = mat4( 1.164,  1.164,  1.164,  -0.07884, 
                                    2.018, -0.391,    0.0,  1.153216,
                                      0.0, -0.813,  1.596,  0.53866, 
                                      0.0,    0.0,    0.0,  1.0);

uniform mediump float centre, range;    

void main()
{
    lowp vec4 srcPixel = texture2D(tex2D, texCoordVarying);
    lowp vec4 yuvPixel = rgbToYuv * srcPixel;

    yuvPixel.r = ((yuvPixel.r - centre) * range) + 0.5;

    gl_FragColor = yuvToRgb * yuvPixel;
}

You'd control that by setting the centre of the range you want to let through (which will be moved to the centre of the output range) and the total range you want to let through (1.0 for the entire range, 0.5 for half the range, etc).

One thing of interest is that I switch from the RGB input space to a YUV colour space for the intermediate adjustment. I do that using a matrix multiplication. I then adjust the brightness channel, and apply another matrix that transforms back from YUV to RGB. To me it made most sense to work in a luma/chroma colour space and from there I picked YUV fairly arbitrarily, though it has the big advantage for ES purposes of being a simple linear transform of RGB space.

I am under the understanding that the curves tool also remaps the brightness, but according to some function f(x) = y, which is monotonically increasing (so, will intersect any horizontal or vertical only exactly once) and is set in the interface as a curve from bottom left to top right somehow.

Because GL ES isn't fantastic with data structures and branching is to be avoided where possible, I'd suggest the best way to implement that is to upload a 256x1 luminance texture where the value at 'x' is f(x). Then you can just map through the secondary texture, e.g. with:

... same as before down to ...
lowp vec4 yuvPixel = rgbToYuv * srcPixel;

yuvPixel.r = texture2D(lookupTexture, vec2(yuvPixel.r, 0.0));

... and as above to convert back to RGB, etc ...

You're using a spare texture unit to index a lookup table, effectively. On iOS devices that support ES 2.0 you get at least eight texture units so you'll hopefully have one spare.

Hue/saturation adjustments are more painful to show because the mapping from RGB to HSV involves a lot of conditionals, but the process is basically the same — map from RGB to HSV, perform the modifications you want on H and S, map back to RGB and output.

Based on a quick Google search, this site offers some downloadable code that includes some Photoshop functions (though not curves or levels such that I can see) and, significantly, supplies example implementations of functions RGBToHSL and HSLToRGB. It's for desktop GLSL, which has a more predefined variables, types and functions, but you shouldn't have any big problems working around that. Just remember to add precision modifiers and supply your own replacements for the absent min and max functions.

Answer 2

For curves photoshop uses bicubic spline interpolation. For a given set of control points you can precalculate all 256 values for each channel and for the master curve. I found that it's easier to store the results as a 256x1 texture and pass it to the shader and then change values of each component:

uniform sampler2D curvesTexture;

vec3 RGBCurvesAdjustment(vec3 color)
{
    return vec3(texture2D(curvesTexture, vec2(color.r, 1.0)).r,
                texture2D(curvesTexture, vec2(color.g, 1.0)).g,
                texture2D(curvesTexture, vec2(color.b, 1.0)).b);
}