How to draw a polygon on specific pixel colors only?

Question

I have drawn multiple different colored polygons on the screen, now I have to draw another polygon of different color, but this polygon should be drawn only on those pixels which have a specific color.

I render each of the different colored polygons at same time in their own "layers", (= one color at a time). They can cover each other; newest layer covers all previous layers. The black color in the image is the "no polygons" area: empty space, and it should ignore that too.

So, basically I just render polygons, and then the N'th (not first) layer of polygons must be masked with the next polygon layer, and nothing else under it should be affected.

Image of the method needed:

What method can I use to achieve this with OpenGL ? I would prefer non-shader solution for this, if possible(?).

The only method I can do currently is to render each of the layers separately into the memory, then go through the pixels myself and combine the layers "manually", but that seems like a very slow method, doable though, but the speed is important here.

Answer 1

To use the stencil buffer for this, what you can do is:

Make sure you request a context that has a stencil buffer, this is windowing system specific so I won't cover it here. Call glGet(GL_STENCIL_BITS) to make sure that you get a sufficient number of bits.

The stencil buffer maintains an integer alongside each pixel, and allows you to modify it as things are drawn. As you draw each layer, set up the stencil buffer to set to a specific value when you draw each layer. You do this with

 //draw layer N
 glEnable(GL_STENCIL_TEST);
 glStencilFunc(GL_ALWAYS, N, -1); 
 glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

Now as you draw each layer, the last polygon that was drawn to the screen also stores it's layer number into the stencil buffer.

At this point when you want to go back and draw your green star, you just tell it to only draw on pixels where the stencil buffer is equal to N.

 //draw only where stencil == N
 glEnable(GL_STENCIL_TEST);
 glStencilFunc(GL_EQUAL, N, -1);
 glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
 drawStar();