Best approach to draw clipped UI elements in OpenGL

Question

I've a complex UI-system which allows a lot of stuff which also can be done with WPF but supports multiple plattforms ( iOS, Android, Windows, ... ). It's not completed yet and now i'm facing the following issue:

My designer wants rotating objects! Rotating objects are far more complex than simple axis aligned ones, which are the reason i can't use glScissor. A little graphic which might help to understand the problem:

You can see that i need to clip the object "Subcontainer" by the bounds of the "Parent Container". As far as i know there are few options:

• Use the stencil buffer, in this case i got a problem because i have objects which are not visible and must also influence the stencil buffer because they are might mask the child object. Also i have to draw each object twice because i need to decrease the stencil buffer when going back in hierarchy.
• Cut the plane ( triangulate; or any other ui model ) which is used to draw the UI-object, this seems to be a lot of afford because they might clipped at different points ( imagine a container in a rotated container in a rotated container... ) and also it's really hard to clip them correctly and it might be a source of performance issues

However both seem to cause a lot of different issues and might be a source for performance leaks. Is there any other way to archive what i want or is there any way to improve the both approaches above?

Answer 1

I ended up with using the Stencil-Buffer, this generates more draw calls than the depth-approach but is much easier to implement.

Before i draw i wrote this code:

if (_Mask)
{
    if (Stage.StencilMaskDepth++ == 0)
        GL.Enable(EnableFlags.STENCIL_TEST);

    GL.ColorMask(false, false, false, false);
    GL.DepthMask(false);
    GL.StencilFunc(StencilFunction.ALWAYS, Stage.StencilMaskDepth, Stage.StencilMaskDepth);
    GL.StencilOp(StencilOp.INCR, StencilOp.INCR, StencilOp.INCR);

    // Draw rectangle
    DrawColor(Colors.Black);

    GL.ColorMask(true, true, true, true);
    GL.DepthMask(true);
    GL.StencilFunc(StencilFunction.EQUAL, Stage.StencilMaskDepth, Stage.StencilMaskDepth);
    GL.StencilOp(StencilOp.KEEP, StencilOp.KEEP, StencilOp.KEEP);
}

After all childs have been drawn this code is called:

if (_Mask)
{
    GL.ColorMask(false, false, false, false);
    GL.DepthMask(false);
    GL.StencilFunc(StencilFunction.ALWAYS, Stage.StencilMaskDepth, Stage.StencilMaskDepth);
    GL.StencilOp(StencilOp.DECR, StencilOp.DECR, StencilOp.DECR);
    // Draw rectangle
    DrawColor(Colors.Black);

    GL.ColorMask(true, true, true, true);
    GL.DepthMask(true);

    if (--Stage.StencilMaskDepth == 0)
        GL.Disable(EnableFlags.STENCIL_TEST);
}

Maybe i going to test some other approaches in a few month but currently this is the easiest to implement.