Trouble understanding the use of dot product in this example?

Question

Usually, I use the dot product of 2 vectors either to find out how perpendicular they are or the cosine of the angle between them. In this shader, a toon shader, the dot product is used on 2 colors and I cannot wrap my head around what exactly this is doing.

uniform vec2 resolution;
uniform sampler2D backBuffer;

void main(void)
{

  vec4 to_gray = vec4(0.3,0.59,0.11,0);
  float x1 = dot(to_gray,  texture2D(backBuffer,vec2((gl_FragCoord.xy
      + vec2(1.0,1.0)) /resolution.xy)).rgba);
  float x0 = dot(to_gray,  texture2D(backBuffer,vec2((gl_FragCoord.xy
      + vec2(-1.0,-1.0)) /resolution.xy)).rgba);
  float x3 = dot(to_gray,  texture2D(backBuffer,vec2((gl_FragCoord.xy
      + vec2(1.0,-1.0)) /resolution.xy)).rgba);
  float x2 = dot(to_gray,  texture2D(backBuffer,vec2((gl_FragCoord.xy
      + vec2(-1.0,1.0)) /resolution.xy)).rgba);
  float edge = (x1 - x0) * (x1 - x0);
  float edge2 = (x3 - x2) * (x3 - x2);
  edge += edge2;
  vec4 color =  texture2D(backBuffer,vec2(gl_FragCoord.xy  / resolution.xy));

    gl_FragColor = max(color - vec4(edge, edge, edge, edge) * 12.0,
        vec4(0,0,0,color.a));
}

Answer 1

The "geometric" scalar (dot) product properties don't really matter in this case. What you have here is an ordinary conversion of some (R, G, B) color to the corresponding grayscale intensity I in accordance with the formula

I = R * 0.30 + G * 0.59 + B * 0.11

(You can learn more about these coefficients here: https://en.wikipedia.org/wiki/Grayscale#Luma_coding_in_video_systems).

As you can immediately see this formula looks like a scalar product of two vectors: one is our color (R, G, B) and the other is (0.30, 0.59, 0.11). So, the author of the code just used the "dot product" function to evaluate this formula for four different color values obtained at four different points: point gl_FragCoord.xy shifted in four different directions (like an x pattern).

In other words, the dot product in this case is not used on "two colors", as you seemed to assume initially. It is used on a color (pixel taken from backBuffer at some coordinates) and a conversion coefficient vector (0.30, 0.59, 0.11) (aptly named to_gray). The latter is not really a color. It just a vector of conversion coefficients. You can think of it as a "color" if you want, but there's not much sense in it.

That's it for the dot product.

Then they do some extra computations to combine the four grayscale values into a single grayscale value. Then they use that grayscale value to modify the original color at point gl_FragCoord.xy (gray values are subtracted from RGB values at gl_FragCoord.xy). The purpose of all this is not fully clear without context.