Writing to the OpenGL Stencil Buffer

Question

I've been reading about the stencil buffer in OpenGL. The basic concept makes sense; a fragment is only drawn if it meets a certain condition after being bitwise ANDed with a value in the stencil buffer. But one thing I don't understand is how you actually write to the stencil buffer. Is there a function that I'm missing here?

P.S. When I say write, I mean specify the specific values in the stencil buffer.

Answer 1

Nobody answered this question and it's a valid question, so more than a year later, here's an answer to your question.

The stencil buffer is theoretically a buffer like the back buffer and the depth buffer. The three of them are written to at the same time (when enabled). You can enable/disable writing to them with specific calls:

• glColorMask(red, green, blue, alpha) - for the back buffer
• glDepthMask(t/f) - for the depth buffer
• glStencilMask(value) - for the stencil buffer

For the depth and stencil buffer, you can specifically enable/disable further with:

• glEnable/glDisable(GL_DEPTH_TEST)
• glEnable/glDisable(GL_STENCIL_TEST)

Any triangle you render to the screen will write to all enabled buffers unless some operation functionality prevents it. For the stencil buffer, these can be set with several functions. Please look up the functionalities on the OpenGL reference pages, but here is a simple example of masking a part of the screen and then rendering only on that masked part of the screen, just to get you started.

glClearColor(0, 0, 0, 1);
glClearStencil(0);
glStencilMask(0xFF);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Do not draw any pixels on the back buffer
glEnable(GL_STENCIL_TEST); // Enables testing AND writing functionalities
glStencilFunc(GL_ALWAYS, 1, 0xFF); // Do not test the current value in the stencil buffer, always accept any value on there for drawing
glStencilMask(0xFF);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE); // Make every test succeed

// ... here you render the part of the scene you want masked, this may be a simple triangle or square, or for example a monitor on a computer in your spaceship ...

glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); // Make sure you will no longer (over)write stencil values, even if any test succeeds
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); // Make sure we draw on the backbuffer again.

glStencilFunc(GL_EQUAL, 1, 0xFF); // Now we will only draw pixels where the corresponding stencil buffer value equals 1

// ... here you render your image on the computer screen (or whatever) that should be limited by the previous geometry ...

glDisable(GL_STENCIL_TEST);

Note that I omitted any depth code on purpose to make sure you see that it has nothing to do with stencilling. If you render 3D geometry, you might need to enable it. You may even need to NOT write a stencil value if a depth-test fails.

Note that it is important that when you render the masking geometry, you set the stencil func to GL_ALWAYS, because otherwise, the current value in the stencil buffer (which was cleared in the example) is tested against whatever was last used and your masking geometry may not even be drawn at all.

So there are no special functions to write to the stencil buffer. I'm not even sure if it can be written to like you can write data directly into the back buffer and the depth buffer video memory, but that's not the way it should be done anyway (because it's terribly slow). The stencil buffer is memory shared with the depth buffer, so it might be possible by changing the parameters of the write functions. I wouldn't count on it working on all video drivers though.

Good luck to anyone who needed this information!