GLSL - Calculate the surface normal given its vertex normal

Question

I want to implement flat shading on OpenGL. I googled it and found this question: How to achieve flat shading with light calculated at centroids?.

I understood the top answer idea and I'm trying to implement it. However, I couldn't figure out how to find the surface normal given the normals of each vertex of the triangle.

Relevant code in vertex shader:

#version 400 core

...

layout(location = 0) in vec4 position;
layout(location = 1) in vec3 normal;

uniform mat4 Mm;
uniform mat3 normalMatrix;

out vec3 vertexN;
out vec3 vertexP;

... 

int main() {
     ...
     vertexN = normalize(normalMatrix * normal);
     vertexP = vec3(Mm * position);
     ...
}

Relevant code in geometry shader:

#version 400 core

...
layout (triangles) in;
layout (triangle_strip, max_vertices=3) out;

in vec3 vertexP[3];
in vec3 vertexN[3];

...

void main(){
    ...
    vec3 centroidPosition(0.0);
    for(int i = 0; i < 3; i++) centroidPosition += vertexP[i];
    centroidPosition/=3.0;

    // calculate surface normal
    
    ...
}

Answer 1

The surface normal can be computed with the Cross product of 2 vectors on the surface. The following code is for counter-clockwise triangles:

vec3 v1 = vertexP[1] - vertexP[0];
vec3 v2 = vertexP[2] - vertexP[0];
vec3 surfNormal = normalize(cross(v1, v2));

If the winding order of the triangles is clockwise, you have to swap v1 and v2:

vec3 surfNormal = normalize(cross(v2, v1));

Use the Dot product to ensure that the orientation of the surface normal correct when mixing the winding order of the triangles:

surfNormal *= sign(dot(surfNormal, vertexN[0]+vertexN[1]+vertexN[2]));