how can I detect if a view matrix is left-handed or right-handed?

Question

I have a camera view matrix which I received from a GL program. I know that this matrix is right-handed since this is the way GL works, but how can I check whether this matrix is right-handed or left-handed programmatically?

For the projection matrix I check if matrix[3][4] (row major) is positive to see if it's left handed. Is this correct?

Thanks.

EDIT:

I have tried the determinant solution, but unfortunately it is not true (at least according to my experiments):

I have used DX9 math functions to test it (to avoid any possible bugs in my code). I have run the following code:

 D3DXVECTOR3 vEye(0,0,0);
 D3DXVECTOR3 vTarget(6,3,0);
 D3DXVECTOR3 vUp(0,0,1);

 D3DXMATRIX matViewLH;
 D3DXMATRIX matViewRH;

 D3DXMatrixLookAtLH(&matViewLH, &vEye, &vTarget, &vUp);
 D3DXMatrixLookAtRH(&matViewRH, &vEye, &vTarget, &vUp);

 float fLHDet = D3DXMatrixDeterminant(&matViewLH);
 float fRHDet = D3DXMatrixDeterminant(&matViewRH);

And the two determinants were equal (both equal to 0.99999994), and obviously had the same sign.

As for my problem - Since I get both the view matrix and projection matrix, and it's relatively easy for me to test whether the projection matrix is LH or RH - I use this information to identify the coordinate system.

Answer 1

You should take the determinant of the matrix. Other things being equal, left-handed and right-handed matrices should have determinants with opposite signs.

It's a little weird because "left-handed" and "right-handed" are pretty much arbitrary conventions. But, matrices that reflect the world like a mirror have negative determinants, so multiplying by such a matrix will change the sign of the determinant.

---

Edit (ref. updated OQ): I suspect the difference between the D3DX*LH() and D3DX*RH() functions is that they support 2 different conventions.

If so, you should be aware that "left-handed" and "right-handed" is not an attribute of each individual matrix, but of how the matrices generated by these sets of functions fit together. If you stick with the *LH() functions, everything should work out properly; similarly with the *RH() functions -- but if you mix them, you will likely get unexpected results.

In any case, neither of these conventions is relevant to GL, which has its own set of conventions, not fully compatible with either of D3D's. For example, GL's convention for Z clipping is different from D3D's; this means that the projection matrix to generate the same view will necessarily differ between the systems.

So, the short answer to your question is "probably neither".

Answer 2

Just bound this same task (just for detecting mirror-transformed objects), found an easy solution, sharing it. If the matrix has a 3x3 part containing 3 perpendicular axes, then you can just make a cross product of axes 1 & 2, then do a scalar product of this result and the remaining 3rd axle. Check if it's negative or positive - you can decide about right/left handedness / object mirrored or not. If I understood the question right...