Dark halo around partial transparent parts of images in opengl

Question

I am routinely finding with OpenGL, when I am trying to draw something on a texture and then later onto another surface, using glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA), I invariably end up with a colored halo (usually a darkish colored band) in what are the semitransparent portions of my source image near its edges. This is evidently caused by issues with making what is essentially a premultiplied alpha caused by that style of blending. For 3d scenes, I typically use a painters algorithm, and for rendering 2d content, I like to draw backmost objects first and overlay additional objects on top of them. It is worth noting that while doing such 2d drawing, I cannot generally rely on features like a z buffer like I can in 3d, which is why it is more of a problem for rendering things like guis than it would be for 3d scene graphs

I think It is worth noting that the above style of blending is actually perfect when the destination buffer happens to already be completely opaque, but if the destination is transparent, then really what is actually needed in this case is glBlendFunc(GL_SRC,GL_ZERO).. since any fully white pixels that are partially transparent in the source, for instance, should remain just as fully saturated with color, and not be "blended" with a background that doesn't actually exist yet, because using glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) would actually render a partially transparent and fully white pixel as a mix between the foreground and background, which is what I want, but not to the extent that alpha information is lost, nor to the extent that the colors are actually changed.

It occurs to me, therefore, the kind of blending function that would probably be perfect would be one that actually blends between what is otherwise caused by glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) and what is otherwise caused by glBlendFunc(GL_SRC,GL_ZERO) depending on the value of DST_ALPHA itself.

Thus, ideally:

Cr = Cs * (1 - Ad) + (Cs * As + Cd * (1 - As)) * Ad
Ar = As * (1 - Ad) + Ad * Ad

This way, if the destination is already opaque, the normal alpha multiplication blends appropriately, while if the destination is transparent or is partially transparent, it will use more of the source image's real color instead of using an alpha-premultiplied form of said colors.

However,there seems to be no apparent way to describe such a blending mechanism to OpenGL. It seems that such issues should hardly be unique to my situation, and it further seems unusual to me that there does not appear to be any way to achieve this... even within a programmable pipeline.

The only workaround that I have found so far is to reverse the drawing order whenever I am drawing to a texture that I will later be drawing elsewhere, and use glBlendFuncSeparate(GL_ONE_MINUS_DST_ALPHA, GL_DST_ALPHA, GL_SRC_ALPHA, GL_DST_ALPHA).

Making a special case for drawing order in the cases I am drawing something on a texture that will later be rendered normally is a compromise, but this is unappealing to me because it lacks reusability, unless the code which is doing the drawing always be made aware of what the type of destination is so that it can reverse its own usual drawing order, which I can complicate it unnnecessarily.

Finally, although reversing the entire scene's drawing order isn't necessarily utterly impossible to do, it requires that a complete list of objects that are going to be displayed in every frame of graphics be determined before even the first object is drawn. The core problem I have with this is that although this may be sustainable when rendering 3D scenes, when drawing things like 2d gui's, the sheer variety of things that are drawn and the way in which they are drawn is simply so vast that there is no way to generalize them into a single object list that can always be traversed in reverse order later. It seems to me like it would require such a radically different view on how to construct 2d scenes in this "backwards" order from what I am accustomed to that I am simply not presently able to even fathom of a maintainable solution.

So... is there another way to do this? Am I missing something? Or do I have to learn a completely different way to construct 2d imagery? I cannot be the only person to have this problem, but I have yet to find any good and clean solution.

Edit:

If there does exist any commonly available extension to OpenGL, by NVIDIA or otherwise, that allows for a blend mode that can do this:

Cr = Cs * (1 - Ad) + (Cs * As + Cd * (1 - As)) * Ad
Ar = As * (1 - Ad) + Ad * Ad

(Where Cs and Cd are the source and destination colors, As and Ad are the source and destination alpha values, and Cr and Ar are the result colors and alpha values), I would really like to know what the extension is called, and how to use it... as long as it can be produced on consumer hardware.

Answer 1

I have written various things compositing translucent 2D overlays onto translucent 3D content and never felt OpenGL was missing something crucial in the area of glBlendFunc / glBlendFuncSeparate (well at least not until you start worrying about gamma-corrected compositing of sRGB content and framebuffers anyway).

I'm mainly suspicious of your use of glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) apparently with premultiplied alpha (sorry, while your question is long, I'm not actually finding it that clear; some images of simple test cases might help). For compositing premultiplied alpha content, I'd expect to see a GL_ONE as one of the glBlendFunc arguments (which one depending on whether you're doing over/under operations).

There's a very good slide deck on blending/compositing (including correct use of premultiplied vs. "straight" colors) here (slides 20-35 the relevant material).

(Despite my claim that what's available already is generally good enough... it's also worth mentioning that NVidia have been adding some new features in this area recently).