GLSL NVidia square artifacts

Question

I have encountered a problem when GLSL shader generates incorrect image on following GPU's:
GT 430
GT 770
GTX 570
GTX 760

But works normally on these:
Intel HD Graphics 2500
Intel HD 4000
Intel 4400
GTX 740M
Radeon HD 6310M
Radeon HD 8850

Shader code is as follows:

bool PointProjectionInsideTriangle(vec3 p1, vec3 p2, vec3 p3, vec3 point)
{
  vec3 n = cross((p2 - p1), (p3 - p1));

  vec3 n1 = cross((p2 - p1), n);
  vec3 n2 = cross((p3 - p2), n);
  vec3 n3 = cross((p1 - p3), n);

  float proj1 = dot((point - p2), n1);
  float proj2 = dot((point - p3), n2);
  float proj3 = dot((point - p1), n3);

  if(proj1 > 0.0)
    return false;
  if(proj2 > 0.0)
    return false;
  if(proj3 > 0.0)
    return false;
  return true;
}

struct Intersection
{
    vec3 point;
    vec3 norm;
    bool valid;
};

Intersection GetRayTriangleIntersection(vec3 rayPoint, vec3 rayDir, vec3 p1, vec3 p2, vec3 p3)
{
    vec3 norm = normalize(cross(p1 - p2, p1 - p3));

    Intersection res;
    res.norm = norm;
    res.point = vec3(rayPoint.xy, 0.0);
    res.valid = PointProjectionInsideTriangle(p1, p2, p3, res.point);
    return res;
}

struct ColoredIntersection
{
    Intersection geomInt;
    vec4 color;
};

#define raysCount 15
void main(void)
{
    vec2 radius = (gl_FragCoord.xy / vec2(800.0, 600.0)) - vec2(0.5, 0.5);

    ColoredIntersection ints[raysCount];

    vec3 randomPoints[raysCount];
    int i, j;


    for(int i = 0; i < raysCount; i++)
    {
        float theta = 0.5 * float(i);
        float phi = 3.1415 / 2.0;
        float r = 1.0;
        randomPoints[i] = vec3(r * sin(phi) * cos(theta),  r * sin(phi)*sin(theta), r * cos(phi));

        vec3 tangent = normalize(cross(vec3(0.0, 0.0, 1.0), randomPoints[i]));
        vec3 trianglePoint1 = randomPoints[i] * 2.0 + tangent * 0.2;
        vec3 trianglePoint2 = randomPoints[i] * 2.0 - tangent * 0.2;

        ints[i].geomInt = GetRayTriangleIntersection(vec3(radius, -10.0), vec3(0.0, 0.0, 1.0), vec3(0.0, 0.0, 0.0), trianglePoint1, trianglePoint2);
        if(ints[i].geomInt.valid)
        {
            float c = length(ints[i].geomInt.point);
            ints[i].color = vec4(c, c, c, 1.0);
        }
    }

    for(i = 0; i < raysCount; i++)
    {
        for(j = i + 1; j < raysCount; j++)
        {
            if(ints[i].geomInt.point.z < ints[i].geomInt.point.z - 10.0)
            {
                gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);
                ColoredIntersection tmp = ints[j];
                ints[j] = ints[i];
                ints[i] = tmp;
            }
        }
    }

    vec4 resultColor = vec4(0.0, 0.0, 0.0, 0.0);
    for(i = 0; i < raysCount + 0; i++)
    {
        if(ints[i].geomInt.valid)
            resultColor += ints[i].color;
    }

    gl_FragColor = clamp(resultColor, 0.0, 1.0);
}

Upd: I have replaced vector normalizations with builtin functions and added gl_FragColor claming just in case.

The code is a simplified version of an actual shader, expected image is:

But what I get is:

Random rotations of the code remove artifacts completely. For example if I change the line

if(ints[i].geomInt.valid) //1

to

if(ints[i].geomInt.valid == true) //1

which apparently should not affect logic in any way or completely remove double cycle that does nothing (marked as 2) artifacts vanish. Please note that the double cycle does nothing at all since condition

if(ints[i].geomInt.point.z < ints[i].geomInt.point.z - 10.0)
{
  gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);
  return;
  ColoredIntersection tmp = ints[j];
  ints[j] = ints[i];
  ints[i] = tmp;
}

Can never be satisfied(left and right sides have index i, not i, j) and there's no NaN's. This code does absolutely nothing yet somehow produces artifacts.

You can test the shader and demo on your own using this project(full MSVS 2010 project + sources + compiled binary and a shader, uses included SFML): https://dl.dropboxusercontent.com/u/25635148/ShaderTest.zip

I use sfml in this test project, but that's 100% irrelevant because the actual project I've enountered this problem does not use this lib.

What I want to know is why these artifacts appear and how to reliably avoid them.

Answer 1

I don't think anything is wrong with your shader. The openGL pipeline renders to a framebuffer. If you make use of that framebuffer before the rendering has completed, you will often get what you have seen. Please bear in mind that glDrawArrays and similar are asynchonous (the function returns before the GPU has finished drawing the vertices.)

The most common use for those square artefacts are when you use the resultant framebuffer as texture which is then use for further rendering.

The OpenGL driver is supposed to keep track of dependencies and should know how to wait for dependencies to be fulfilled.

If you are sharing a framebuffer across threads, however, all bets are off, you then might need to make use of things like a fence sync (glFenceSync) to ensure that that one thread waits for rendering that is taking place on another thread.

As a workaround, you might find calling glFinish or even glReadPixels (with one pixel) sorts the issue out.

Please also bear in mind that this problem is timing related and simplifying a shader might very well make the issue go away.

Answer 2

If anyone's still interested I asked this question on numerous specialized sites including opengl.org and devtalk.nvidia.com. I did not receive any concrete answer on what's wrong with my shader, just some suggestions how to work around my problem. Like use if(condition == true) instead of if(condition), use as simple algorithms as possible and such. In the end I've chosen one of the easiest rotations of my code that gets rid of the problem: I just replaced

struct Intersection  
{  
    vec3 point;  
    vec3 norm;  
    bool valid;  
};  

with

struct Intersection  
{  
    bool valid;  
    vec3 point;  
    vec3 norm;  
};  

There were numerous other code rotations that made the artifacts disappear, but I've chosen this one because I was able to test in on most other systems I had trouble with before.