OpenGL - Question about the usage of glDepthMask

Question

I have rendered an objectA in a scene as follows. The scene has many other objects too.

void Draw()
{    
    if( glIsList( displayListID ) )
    {
        glPushAttrib( GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT|GL_ENABLE_BIT );

        glEnable( GL_BLEND );
        glEnable( GL_DEPTH_TEST );
        //glDepthMask( GL_FALSE );
        glBlendFunc( GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA );

        glEnable( GL_LINE_SMOOTH );
        glEnable( GL_POINT_SMOOTH );
        glEnable( GL_POLYGON_SMOOTH );

        glMatrixMode( GL_MODELVIEW ); 

        color.setAlpha(alpha); // set alpha transparent of this objectA
        glCallList( displayListID );

        //glDepthMask( GL_TRUE );
        glDisable( GL_BLEND );  

        glPopAttrib();
    }
}

Now here is the problem,

As shown, I comment out two lines //glDepthMask( GL_FALSE ); //glDepthMask( GL_TRUE );

the scene renders the objectA and other objects correctly in depth. However, the modification of the objectA alpha doesn't work anymore (i.e. color.setAlpha(alpha) ).

If I uncomment the above two lines, then alpha modification is back to work. However, the depth rendering is NOT correct. In other words, sometimes, the objectA should be behind other objects but it looks like the objectA is in front of all objects.

How do I fix this problem?

Thank you

Answer 1

1. Turn on the depth mask glDepthMask( GL_TRUE )
2. Draw all opaque objects, in any order
3. Turn off the depth mask glDepthMask( GL_FALSE )
4. Turn on a BLEND_MODE
5. Draw translucent objects sorted from furthest away to nearest

Why do you do this?

There are 2 buffers you need to worry about: the depth buffer and the color buffer. These buffers are really just big 2d arrays, each the width x height of your screen.

The color buffer naturally is going to hold the final coloration of each pixel. There is one entry in the color buffer per screen pixel. The depth buffer, is like the color buffer in that there is one entry per screen pixel, but it is used for something different. Entries in the depth buffer are a measure of "how close" each colored pixel really is.

If you render 1 triangle, that is far away from the camera, it generates a set of colors and depth values for each pixel it "wants" to cover on the screen. Say you then render another poly that is closer, it also will generate a set of values for the depth and color buffers. Now, there is a sort of "contest" at pixel coloration time where the "further away" fragments (large depth buffer values) are discarded, and only the closest fragments are kept. The closer fragments end up coloring the pixel you had. (When two polygons are nearly overlapping, Z-fighting can occur)

Start by rendering the objects in your scene with the depth mask on. This means every shape you render, when its pixels get colored, the depth buffer gets updated with the "winner" of the contest.

Then, you 3) glDepthMask( GL_FALSE ) turns off the depth buffer for writing, 4) turn on blending, 5) render translucent shapes from furthest to nearest. Seems weird, huh?

When you turn off the depth mask, and render the translucent shapes, OpenGL will still read the depth buffer to determine which fragments to throw away (i.e. if your translucent shape is behind an already rendered solid shape, then you throw that translucent shape's fragments away). But it will not write to the depth buffer, so if a translucent shape is really really close to the eye (say like a translucent windshield), those windshield fragments do not prevent other fragments that are actually further away from being drawn. This is important, because if your windshield is right in front of you and you render it translucent, and you let the windshield fragments update the depth buffer then you will see nothing else in your scene except the windshield, even though there are shapes behind it, because OpenGL will think "Hey, that windshield is the only thing the user should see, due to these depth buffer readings, so I won't bother rendering anything further away than this windshield, then." Turning off the depth mask is a way of "tricking" OpenGL into "not knowing" there are very close, but translucent, fragments.

Answer 2

One possible solution: set glDepthMask to GL_TRUE at all times and draw all your non-transparent objects first (in any order, just as you seem to do it now), then draw all your (semi-)transparent objects sorted from back to front.

In some cases, if you don't care about the order in which your (semi-)transparent objects are drawn and you only want the other, opaque objects to "shine through", you can skip sorting your (semi-)transparent objects.

Answer 3

Minor addition to other responses:

Even though visually it makes no difference what order you draw your opaque objects in, you should nevertheless sort them front-to-back to avoid over-draw and take advantage of the depth buffer to avoid pointless pixel shader invocations. This can have a significant performance impact. And even if sorting is slow, you can still bin your objects into near/medium/far groups and draw those in an explicit order.

For instance, in a 3D game, you always want to draw the sky last. Right before that, the landscape. Before that, distant objects. But start with the player or gun. The most stuff you draw first, that's closer, the less stuff you'll draw then overdraw. It's not always possible to do this with overlapping objects, but it's usually good enough.