GLSL point inside box test

Question

Below is a GLSL fragment shader that outputs a texel if the given texture coord is inside a box, otherwise a color is output. This just feels silly and the there must be a way to do this without branching?

uniform sampler2D texUnit;

varying vec4 color;
varying vec2 texCoord;

void main() {
  vec4 texel = texture2D(texUnit, texCoord);
  if (any(lessThan(texCoord, vec2(0.0, 0.0))) ||
      any(greaterThan(texCoord, vec2(1.0, 1.0))))
    gl_FragColor = color;
  else
    gl_FragColor = texel;
}

Below is a version without branching, but it still feels clumsy. What is the best practice for "texture coord clamping"?

uniform sampler2D texUnit;

varying vec4 color;
varying vec4 labelColor;
varying vec2 texCoord;

void main() {
  vec4 texel = texture2D(texUnit, texCoord);
  bool outside = any(lessThan(texCoord, vec2(0.0, 0.0))) ||
                 any(greaterThan(texCoord, vec2(1.0, 1.0)));
  gl_FragColor = mix(texel*labelColor, color,
                     vec4(outside,outside,outside,outside));
}

I am clamping texels to the region with the label is -- the texture s & t coordinates will be between 0 and 1 in this case. Otherwise, I use a brown color where the label ain't.

Note that I could also construct a branching version of the code that does not perform a texture lookup when it doesn't need to. Would this be faster than a non-branching version that always performed a texture lookup? Maybe time for some tests...

Answer 1

Use step function to avoid branching and get the best performance:

// return 1 if v inside the box, return 0 otherwise
float insideBox(vec2 v, vec2 bottomLeft, vec2 topRight) {
    vec2 s = step(bottomLeft, v) - step(topRight, v);
    return s.x * s.y;   
}

float insideBox3D(vec3 v, vec3 bottomLeft, vec3 topRight) {
    vec3 s = step(bottomLeft, v) - step(topRight, v);
    return s.x * s.y * s.z; 
}

void main() {
    vec4 texel = texture2D(texUnit, texCoord);

    float t = insideBox(v_position, vec2(0, 0), vec2(1, 1));
    gl_FragColor = t * texel + (1 - t) * color;
}

Answer 2

Based on damphat's answer, I've implemented a function for a smooth transition between inside and outside the rectangle:

float inside_rectangle_smooth(vec2 p, vec2 bottom_left, vec2 top_right, float transition_area)
{
    vec2 s = smoothstep(bottom_left, bottom_left + vec2(transition_area), p) -
             smoothstep(top_right - vec2(transition_area), top_right, p);
    return(s.x * s.y);
}

Use the "transition_area" parameter to tune the size of the transition area between inside and outside the rectangle. The transition is faded into the rectangle, not out of it. Also make sure the "transition_area" parameter is smaller than the distance between "bottom_left" and "top_right" (in each dimension).
I'm successfully using this function to fade shadows in my engine (using shadow map coordinates).

Demonstration:

The above image is produced by calling:
inside_rectangle_smooth(v_texture_coordinate, vec2(0.0), vec2(1.0), 0.1)

Answer 3

I just cam across this solution. Via https://github.com/stackgl/shader-school

bool inBox(highp vec2 lo, highp vec2 hi, highp vec2 p) {
    bvec4 b = bvec4(greaterThan(p, lo), lessThan(p, hi));
    return all(b);
}