Converting glm::lookat matrix to quaternion and back

Question

I am using glm to create a camera class, and I am running into some problems with a lookat function. I am using a quaternion to represent rotation, but I want to use glm's prewritten lookat function to avoid duplicating code. This is my lookat function right now:

void Camera::LookAt(float x, float y, float z) {
    glm::mat4 lookMat = glm::lookAt(position, glm::vec3(x, y, z), glm::vec3(0, 1, 0));
    rotation = glm::toQuat(lookMat);
}

However when I call LookAt(0.0f,0.0f,0.0f), my camera is not rotated to that point. When I call glm::eulerangles(rotation) after the lookat call, I get a vec3 with the following values: (180.0f, 0.0f, 180.0f). position is (0.0f,0.0f,-10.0f), so I should not have any rotation at all to look at 0,0,0. This is the function which builds the view matrix:

glm::mat4 Camera::GetView() {
    view = glm::toMat4(rotation) * glm::translate(glm::mat4(), position);
    return view;
}

Why am I not getting the correct quaternion, and how can I fix my code?

Answer 1

I ran into something similar, the short answer is your lookMat might need to be inverted/transposed, because it is a camera rotation (at least in my case), as opposed to a world rotation. Rotating the world would be a inverse of a camera rotation.

I have a m_current_quat which is a quaternion that stores the current camera rotation. I debugged the issue by printing out the matrix produced by glm::lookAt, and comparing with the resulting matrix that I get by applying m_current_quat and a translation by m_camera_position. Here is the relevant code for my test.

void PrintMatrix(const GLfloat m[16], const string &str)
{
    printf("%s:\n", str.c_str());

    for (int i=0; i<4; i++)
    {
        printf("[");
        //for (int j=i*4+0; j<i*4+4; j++)   // row major, 0, 1, 2, 3
        for (int j=i+0; j<16; j+=4) // OpenGL is column major by default, 0, 4, 8, 12
        {
            //printf("%d, ", j);            // print matrix index
            printf("%.2f, ", m[j]);

        }
        printf("]\n");
    }
    printf("\n");
}

void CameraQuaternion::SetLookAt(glm::vec3 look_at)
{
    m_camera_look_at = look_at;

    // update the initial camera direction and up
    //m_initial_camera_direction = glm::normalize(m_camera_look_at - m_camera_position);
    //glm::vec3 initial_right_vector = glm::cross(m_initial_camera_direction, glm::vec3(0, 1, 0));
    //m_initial_camera_up = glm::cross(initial_right_vector, m_initial_camera_direction);

    m_camera_direction = glm::normalize(m_camera_look_at - m_camera_position);
    glm::vec3 right_vector = glm::cross(m_camera_direction, glm::vec3(0, 1, 0));
    m_camera_up = glm::cross(right_vector, m_camera_direction);


    glm::mat4 lookat_matrix = glm::lookAt(m_camera_position, m_camera_look_at, m_camera_up);

    // Note: m_current_quat quat stores the camera rotation with respect to the camera space
    // The lookat_matrix produces a transformation for world space, where we rotate the world
    // with the camera at the origin
    // Our m_current_quat need to be an inverse, which is accompolished by transposing the lookat_matrix
    // since the rotation matrix is orthonormal.
    m_current_quat = glm::toQuat(glm::transpose(lookat_matrix));    

    // Testing: Make sure our model view matrix after gluLookAt, glmLookAt, and m_current_quat agrees
    GLfloat current_model_view_matrix[16];              

    //Test 1: gluLookAt
    gluLookAt(m_camera_position.x, m_camera_position.y, m_camera_position.z,
                m_camera_look_at.x, m_camera_look_at.y, m_camera_look_at.z,
                m_camera_up.x, m_camera_up.y, m_camera_up.z);       
    glGetFloatv(GL_MODELVIEW_MATRIX, current_model_view_matrix);                        
    PrintMatrix(current_model_view_matrix, "Model view after gluLookAt");   

    //Test 2: glm::lookAt
    lookat_matrix = glm::lookAt(m_camera_position, m_camera_look_at, m_camera_up);
    PrintMatrix(glm::value_ptr(lookat_matrix), "Model view after glm::lookAt");

    //Test 3: m_current_quat
    glLoadIdentity();
    glMultMatrixf( glm::value_ptr( glm::transpose(glm::mat4_cast(m_current_quat))) );
    glTranslatef(-m_camera_position.x, -m_camera_position.y, -m_camera_position.z);
    glGetFloatv(GL_MODELVIEW_MATRIX, current_model_view_matrix);                        
    PrintMatrix(current_model_view_matrix, "Model view after quaternion transform");    

    return;
}

Hope this helps.