Z-fighting after depth prepass on GTX 980

Question

I'm implementing a depth prepass in OpenGL. On an Intel HD Graphics 5500, this code works fine but on a Nvidia GeForce GTX 980 it doesn't (the image below shows the resulting z-fighting). I'm using the following code to generate the image. (Everything irrelevant to the problem is omitted.)

// ----------------------------------------------------------------------------
// Depth Prepass
// ----------------------------------------------------------------------------

glEnable(GL_DEPTH_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthFunc(GL_LESS);
glDepthMask(GL_TRUE);

glUseProgam(program1); // The problem turned out to be here!

renderModel(...);

// ----------------------------------------------------------------------------
// Scene Rendering
// ----------------------------------------------------------------------------

glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_LEQUAL);
glDepthMask(GL_FALSE);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

glUseProgam(program2); // The problem turned out to be here!

renderModel(...);

It seems like the glDepthFunc is not changed to GL_LEQUAL. However, when I step through the GL calls in RenderDoc, glDepthFunc is set correnctly.

Does this sound like a driver bug or do you have suggestions what I could be doing wrong? When this is a driver bug, how can I implement a depth prepass anyway?

Answer 1

When using a different shader program for the depth prepass it must be explicitly assured that this program generates the same depth values (although called on the same geometry) as the program for the main pass. This is done by using the invariant qualifier on gl_Position.

Variance explained by the GLSL Specification 4.4:

In this section, variance refers to the possibility of getting different values from the same expression in different programs. For example, say two vertex shaders, in different programs, each set gl_Position with the same expression in both shaders, and the input values into that expression are the same when both shaders run. It is possible, due to independent compilation of the two shaders, that the values assigned to gl_Position are not exactly the same when the two shaders run. In this example, this can cause problems with alignment of geometry in a multi-pass algorithm.

The qualifier is used as follows in this case:

invariant gl_Position;

This line guarantees that gl_Position is computed by the exact expression that was given in the shader without any optimization as this would change the operations and therefore quite likely change the result in some minor way.

In my concrete case, an assignment was the source of the problem. The Vertex Shader of the program for the main pass contained the following lines:

fWorldPosition = ModelMatrix*vPosition; // World position to the fragment shader
gl_Position = ProjectionMatrix*ViewMatrix*fWorldPosition;

The Vertex Shader of the program for the prepass computed gl_Position in one expression:

gl_Position = ProjectionMatrix*ViewMatrix*ModelMatrix*vPosition;

By changing this into:

vec4 worldPosition = ModelMatrix*vPosition;
gl_Position = ProjectionMatrix*ViewMatrix*worldPosition;

I solved the problem.