Correctly transforming a surface normal

Question

According to the OpenGL Red Book,Appendix F, a regular 3D transformation matrix M can be used to calculate the action on a normal vector as:

normalTransformed = transpose(inverse(M)) * normal

However, while the orthogonal plane associated with the transformed normal is indeed parallel to the transformed surface, it can happen that the transformed normal vector itself is pointing in the opposite direction of what I would expect, i.e., 'into' the surface rather than 'out of' the surface.

If I want the normalTransformed to point in the correct direction (i.e., the same direction as it points to when the surface to which it is attached is not transformed), how should I do that, mathematically?

EXAMPLE

Suppose my surface normal is (0,0,1), and my transform is a translation by 10 in the Z direction. The transformation matrix M is then:

1 0 0 0
0 1 0 0
0 0 1 10
0 0 0 1

The transpose(inverse(M)) is then:

1 0 0 0
0 1 0 0
0 0 1 0
0 0 -10 1

Applied to the surface normal (0,0,1), i.e., (0,0,1,1) in homogeneous coordinates, this gives:

normalTransformed = (0, 0, 1, -9)

Back from homogeneous coordinates:

(0, 0, -1/9)

Normalizing to length 1:

(0, 0, -1)

Which points to the opposite direction compared to the original normal vector (0, 0, 1).

Answer 1

Applied to the surface normal (0,0,1), i.e., (0,0,1,1) in homogeneous coordinates

OK, stop right there.

If you're going to look at surface normals as homogeneous coordinates, you use a zero as the W component, not a 1. Now, you would probably realize fairly quickly that you can't divide by zero, but that's also why you don't do homogeneous math on normals.

A normal is not a position; it is a direction. Directions do not have a position, so translating them is meaningless. A homogeneous position with a W=0 represents a "position" infinitely far away (which is why you can't divide them). A position infinitely far away is infinitely far away from every finite point.

And therefore, a position at infinity is a direction: it doesn't change direction no matter what (finite) position you look at it from.

Now, if you have a 4x4 matrix and need to transform a normal by it, you only use W=0 because it makes the math work out. It gets rid of the translation component of the matrix. The post-transform W component should be completely ignored.

Therefore, after transforming, you get this:

normalTransformed = (0, 0, 1, -9)

Which after ignoring the W component, becomes:

normalTransformed = (0, 0, 1)

---

It is far more likely that your normal is not actually pointed in the right direction to begin with. Of course, in the absence of code and data, little more can be said, but the math works assuming the inputs are legitimate.

Also, don't do the inverse/transpose in the shader. Do it on the CPU and pass the resulting matrix to the shader.