Shadow mapping shader

Question

I've got a shader that implements shadow mapping like this:

#version 430 core

out vec4 color;

in VS_OUT {
    vec3 N;
    vec3 L;
    vec3 V;
    vec4 shadow_coord;
} fs_in;

layout(binding = 0) uniform sampler2DShadow shadow_tex;

uniform vec3 light_ambient_albedo = vec3(1.0);
uniform vec3 light_diffuse_albedo = vec3(1.0);
uniform vec3 light_specular_albedo = vec3(1.0);

uniform vec3 ambient_albedo = vec3(0.1, 0.1, 0.2);
uniform vec3 diffuse_albedo = vec3(0.4, 0.4, 0.8);
uniform vec3 specular_albedo = vec3(0.0, 0.0, 0.0);
uniform float specular_power = 128.0;

void main(void) {
    //color = vec4(0.4, 0.4, 0.8, 1.0);

    //normalize
    vec3 N = normalize(fs_in.N);
    vec3 L = normalize(fs_in.L);
    vec3 V = normalize(fs_in.V);

    //calculate R
    vec3 R = reflect(-L, N);

    //calcualte ambient
    vec3 ambient = ambient_albedo * light_ambient_albedo;

    //calculate diffuse
    vec3 diffuse = max(dot(N, L), 0.0) * diffuse_albedo * light_diffuse_albedo;

    //calcualte spcular
    vec3 specular = pow(max(dot(R, V), 0.0), specular_power) * specular_albedo * light_specular_albedo;

    //write color
    color = textureProj(shadow_tex, fs_in.shadow_coord) * vec4(ambient + diffuse + specular, 0.5);
    //if in shadow, then multiply color by 0.5 ^^, except alpha
}

What I want to do is to check first if the fragment is indeed in the shadow, and only then change the color (halve it, such that it becomes halfway between fully black and original color).

However how to check if the textureProj(...) result is indeed in shadow, as far as I know it returns a normalized float value.

Would something like textureProj(...) > 0.9999 suffice already? I know that it can returns values other than zero or one if you are using multisampling and I'd like behaviour that will not just break at one point.

The outputting vertex shader:

#version 430 core

layout(location = 0) in vec4 position;

layout(location = 0) uniform mat4 model_matrix;
layout(location = 1) uniform mat4 view_matrix;
layout(location = 2) uniform mat4 proj_matrix;
layout(location = 3) uniform mat4 shadow_matrix;

out VS_OUT {
    vec3 N;
    vec3 L;
    vec3 V;
    vec4 shadow_coord;
} vs_out;

uniform vec4 light_pos = vec4(-20.0, 7.5, -20.0, 1.0);

void main(void) {
    vec4 local_light_pos = view_matrix * light_pos;
    vec4 p = view_matrix * model_matrix * position;

    //normal
    vs_out.N = vec3(0.0, 1.0, 0.0);

    //light vector
    vs_out.L = local_light_pos.xyz - p.xyz;

    //view vector
    vs_out.V = -p.xyz;

    //light space coordinates
    vs_out.shadow_coord = shadow_matrix * position;

    gl_Position = proj_matrix * p;
}

Note that the fragment shader is for terrain, and the vertex shader is for the floor, so there might be minor inconsistencies between the two, but they should be non relevant.

shadow_matrix is an uniform passed in as bias_matrix * light_projection_matrix * light_view_matrix * light_model_matrix.

Answer 1

textureProj (...) does not return a normalized floating-point value. It does return a single float if you use it on a sampler<1D|2D|2DRect>Shadow, but this value represents the result of a depth test. 1.0 = pass, 0.0 = fail.

Now, the interesting thing to note here, and the reason returning a float for a shadow sampler is meaningful at all has to do with filtering the shadow map. If you use a GL_LINEAR filter mode on the shadow map together with a shadow sampler, GL will actually pick the 4 closest texels in the shadow map and perform 4 independent depth tests.

Each depth test still has a binary result, but GL will return a weighted average of the result of all 4 tests (based on distance from the ideal sample location). So if you use GL_LINEAR in conjunction with a shadow sampler, you will have a value that lies somewhere in-between 0.0 and 1.0 representing the average occlusion for the 4 nearest depth samples.

I should point out that your use of textureProj (...) looks potentially wrong to me. The coordinate it uses is a 4D vector consisting of (s,t,r) [projected coordinates] and (q) [depth value to test]. I do not see anywhere in your code where you are assigning q a depth value. If you could edit your question to include the vertex/geometry shader that is outputting shadow_coord, that would help.