Projection Mapping with Kinect and OpenGL

Question

Im currently using a JavaCV software called procamcalib to calibrate a Kinect-Projector setup, which has the Kinect RGB Camera as origin. This setup consists solely of a Kinect RGB Camera (Im roughly using the Kinect just as an ordinary camera at the moment) and one Projector. This calibration software uses LibFreenect (OpenKinect) as the Kinect Driver.

Once the software completes its process, it will give me the intrinsics and extrinsics parameters of both the camera and the projector, which are being thrown at an OpenGL software to validate the calibration and is where a few problems begins. Once the Projection and Modelview are correctly set, I should be able to fit what is seen by the Kinect with what is being projected, but in order to achieve this I have to do a manual translation in all 3 axis and this last part isnt making any sense to me! Could you guys please help me sorting this out? The SDK used to retrieve Kinect data is OpenNI (not the latest 2.x version, it should be 1.5.x)

I'll explain exactly what Im doing to reproduce this error. The calibration parameters is used as follows:

The Projection matrix is set as ( based on http://sightations.wordpress.com/2010/08/03/simulating-calibrated-cameras-in-opengl/ ):

r = width/2.0f;         l = -width/2.0f; t = height/2.0f;        b = -height/2.0f;  alpha = fx;      beta = fy; xo    = cx;      yo   = cy;  X = kinectCalibration.c_near + kinectCalibration.c_far; Y = kinectCalibration.c_near*kinectCalibration.c_far;  d = kinectCalibration.c_near - kinectCalibration.c_far;   float* glOrthoMatrix = (float*)malloc(16*sizeof(float));  glOrthoMatrix[0] = 2/(r-l); glOrthoMatrix[4] = 0.0f;        glOrthoMatrix[8] = 0.0f;        glOrthoMatrix[12] = (r+l)/(l-r); glOrthoMatrix[1] = 0.0f;    glOrthoMatrix[5] = 2/(t-b);     glOrthoMatrix[9] = 0.0f;        glOrthoMatrix[13] = (t+b)/(b-t); glOrthoMatrix[2] = 0.0f;    glOrthoMatrix[6] = 0.0f;        glOrthoMatrix[10] = 2/d;        glOrthoMatrix[14] = X/d; glOrthoMatrix[3] = 0.0f;    glOrthoMatrix[7] = 0.0f;        glOrthoMatrix[11] = 0.0f;       glOrthoMatrix[15] = 1; printM( glOrthoMatrix, 4, 4, true, "glOrthoMatrix" );   float* glCameraMatrix = (float*)malloc(16*sizeof(float));  glCameraMatrix[0] = alpha;  glCameraMatrix[4] = skew;   glCameraMatrix[8] = -xo;    glCameraMatrix[12] = 0.0f; glCameraMatrix[1] = 0.0f;   glCameraMatrix[5] = beta;   glCameraMatrix[9] = -yo;    glCameraMatrix[13] = 0.0f; glCameraMatrix[2] = 0.0f;   glCameraMatrix[6] = 0.0f;   glCameraMatrix[10] = X;     glCameraMatrix[14] = Y; glCameraMatrix[3] = 0.0f;   glCameraMatrix[7] = 0.0f;   glCameraMatrix[11] = -1;    glCameraMatrix[15] = 0.0f;  float* glProjectionMatrix = algMult( glOrthoMatrix, glCameraMatrix ); 

And the Modelview matrix is set as:

proj_loc = new Vec3f(   proj_RT[12], proj_RT[13], proj_RT[14] );     proj_fwd = new Vec3f(   proj_RT[8],  proj_RT[9],  proj_RT[10] ); proj_up  = new Vec3f(   proj_RT[4],  proj_RT[5],  proj_RT[6]  ); proj_trg = new Vec3f(   proj_RT[12] + proj_RT[8],                          proj_RT[13] + proj_RT[9],                          proj_RT[14] + proj_RT[10] );  gluLookAt( proj_loc[0], proj_loc[1], proj_loc[2],            proj_trg[0], proj_trg[1], proj_trg[2],            proj_up[0],  proj_up[1],  proj_up[2] ); 

And finally the camera is displayed and moved around with:

glPushMatrix(); glTranslatef(translateX, translateY, translateZ);  drawRGBCamera(); glPopMatrix(); 

where the translation values are manually adjusted with the keyboard until I have a visual match (I'm projecting on the calibration board what the Kinect-rgb camera is seeing, so I manually adjust the opengl-camera until the projected pattern matches the printed pattern).

My question here is WHY do I have to make this manual adjustment? The modelview and projection setup should take care of it.

I was also wandering if there are any problems when switching drivers like that, since OpenKinect is used for calibration and OpenNI for validation. This came at mind when researching another popular calibration tool called RGBDemo, where it says that if using LibFreenect backend a Kinect calibration is needed.

So, will a calibration go wrong if made with a driver and displayed with another?

Does anyone think it'll be easier to achieve success if this is done with OpenCV rather than OpenGL ?

JavaCV Reference: https://code.google.com/p/javacv/
Procamcalib "short paper": http://www.ok.ctrl.titech.ac.jp/~saudet/research/procamcalib/
Procamcalib source code: https://code.google.com/p/javacv/source/browse?repo=procamcalib
RGBDemo calibration Reference: http://labs.manctl.com/rgbdemo/index.php/Documentation/Calibration

I can upload more things if necessary, just let me know what you guys need to be able to help me out :)

Answer 1

I'm the author of the article you linked to, and I think I can help.

The problem is in how you're setting your modelview matrix. You're using the third column of proj_RT as the camera's position when you call gluLookAt(), but it isn't the camera's position, it's the position of the world origin in camera coordinates. I wrote an article for my new blog that might help clear this up. It describes three different (equivalent) ways of interpreting the extrinsic matrix, with WebGL demos of each:

http://ksimek.github.io/2012/08/22/extrinsic/

If you must use gluLookAt, this article will show you how, but its much simpler to just call glLoadMatrix(proj_RT).

tl;dr: replace gluLookAt() with glLoadMatrix(proj_RT)