OpenGL - Local Up and Right From Matrix 4x4?

Question

I have a transform matrix 4x4 built in the following way:

glm::mat4 matrix;
glm::quat orientation = toQuaternion(rotation);
matrix*= glm::mat4_cast(orientation);
matrix= glm::translate(matrix, position);
matrix= glm::scale(matrix, scale);

orientation = toQuaternion(localRotation);
matrix*= glm::mat4_cast(orientation);
matrix= glm::scale(matrix, localScale);

where rotation, position, scale, localRotation, and localScale are all vector3's. As per the advice of many different questions on many different forums, it should be possible to fish local directions out of the resulting matrix like so:

right   = glm::vec3(matrix[0][0], matrix[0][1], matrix[0][2]);
up      = glm::vec3(matrix[1][0], matrix[1][1], matrix[1][2]);
forward = glm::vec3(matrix[2][0], matrix[2][1], matrix[2][2]);

where all these directions are normalized. I then use these directions to get a view matrix like so:

glm::lookAt(position, position + forward, up);

It all works great - EXCEPT: when I'm flying around a scene, the right and up vectors are totally erroneous. The forward direction is always exactly as it should be. When I'm at 0, 0, -1 looking at 0,0,0, all the directions are correct. But when I look in different directions (except for the inverse of looking at the origin from 0,0,-1), right and up are inconsistent. They aren't world vectors, nor are they local vectors. They seem to be almost random. Where am I going wrong? How can I get consistent local up and local right vectors from the 4x4 transform matrix?

Answer 1

matrix is a local-to-world transformation, whereas a view matrix is world-to-local, which means that in this case matrix is the inverse of the view matrix. The code you currently have only works for a view matrix. Simply exchange the rows and columns:

right   = glm::vec3(matrix[0][0], matrix[1][0], matrix[2][0]);
up      = glm::vec3(matrix[0][1], matrix[1][1], matrix[2][1]);
forward = glm::vec3(matrix[0][2], matrix[1][2], matrix[2][2]);

A possible reason that it works for (0, -1, 0) is because the rotation/scaling part of the view matrix looks like:

1 0 0
0 0 1
0 1 0

The corresponding part of matrix is the inverse of the above, which is of course identical. Try it with another direction.