How to rotate an object according to it's orientation

Question

Similar question for WebGL: Rotate object around world axis .

I need to rotate an object in a way that the user should be able to move it with the mouse, like if he's dragging it. The problem is that glRotatef just rotates the object without taking account of it's orientation. For WebGL the solution was to use quaternions, but I guess that there aren't quaternions in OpenGL.

This is how I achieve a rotation for now:

// rotation 2D GLfloat C-vector, posX, posY GLfloat's
void mouse(int button, int state, int x, int y) {
    if(button== GLUT_LEFT_BUTTON) {
        if(state== GLUT_DOWN) 
        {
            posX=x;
            posY= y;
            onEvent= true;
        }
        else if(state== GLUT_UP) 
        {
            GLfloat dx= x-posX;
            GLfloat dy= y-posY;
            rotation[0]= -dy/5;
            rotation[1]= dx/5;
            onEvent= false;
            glutPostRedisplay();
        }
    }
}

Then I handle the rotation in the display function:

glPushMatrix();
glRotatef(rotation[0],1,0,0);
glRotatef(rotation[1],0,1,0);
// Draw the object
glPopMatrix();

It kinda works, but like I said it should like if the user is able to drag the object to rotate it. Instead if for example the object is rotated of 90 degrees around the X axis, when the user drags the mouse horizontally to make it rotate around the Y axis, it rotates in the inverse direction. I need an idea here, how could I do that?

Edit

I tried to use glMultMatrixf, but the object doesn't rotate correctly: it gets scaled instead of rotating, this is the code I've edited in the mouse function:

// Global variables:
// GLfloat xRotationMatrix[4][4];
// GLfloat yRotationMatrix[4][4];
else if(state== GLUT_UP && onEvent)
{
    GLfloat dx= (x-posX)/(180.0*5)*2.0*M_PI;
    GLfloat dy= (y-posY)/(180.0*5)*2.0*M_PI;
    // Computing rotations
    double cosX= cos(dx);
    double sinX= sin(dy);
    double cosY= cos(dy);
    double sinY= sin(dy);
    // x axis rotation
    xRotationMatrix[1][1]+= cosY;
    xRotationMatrix[1][2]+=-sinY;
    xRotationMatrix[2][2]+= sinY;
    xRotationMatrix[2][2]+= cosY;
    // y axis rotation
    yRotationMatrix[0][0]+= cosX;
    yRotationMatrix[0][2]+= sinX;
    yRotationMatrix[2][0]+= -sinX;
    yRotationMatrix[2][2]+= cosX;

    onEvent= false;
    glutPostRedisplay();
}

Then in the display function:

glPushMatrix();
glMultMatrixf((const GLfloat*)xRotationMatrix);
glMultMatrixf((const GLfloat*)yRotationMatrix);
glutSolidTeapot(10);
glPopMatrix();

This is the non rotated teapot:

If I drag the mouse horizontally to rotate the teapot around the y axis, instead of the rotation this is what I get:

Answer 1

First of all a bit of algebra. Let v be a vector, M your current modelview matrix, and R the matrix associated with a glRotate command. Then, if you use glRotate, what you get is:

M * R * v

That means you are rotating around object axes. You want to rotate around the world axes, that is:

R * M * v

See the difference? Unfortunately GL doesn't have a MatrixPreMult function.

In modern OpenGL we don't use the matrix stack anymore, in fact while working with shaders we manually pass the transformation matrices to the GL program. What (most) people do is write/use an external vector algebra library (like Eigen).

One possible (untested) workaround which uses only the old deprecated GL stuffs may be something like this:

void rotate(float dx, float dy)
{
     //assuming that GL_MATRIX_MODE is GL_MODELVIEW
     float oldMatrix[4][4];
     glGetFloatv(GL_MODELVIEW_MATRIX,oldMatrix);
     glLoadIdentity();
     glRotatef(-dy,1,0,0);
     glRotatef(dx,0,1,0);
     glMultMatrixf(oldMatrix);
}

And you put this code in your mouse function, not in the draw routine. You can use this trick by keeping the view matrix in the GL matrix stack, then pushing/popping everytime you have to draw an object. I wouldn't recommend something like that in a large project.

---

Notice also that if you invert the order of the two glRotate calls in the code above you can get slightly different results, expecially if dx and dy are not small. This code might be slightly better:

float angle = sqrt(dx*dx+dy*dy)*scalingFactor;
glRotate(angle,-dy,dx,0);