OpenCV + OpenGL + Qt

Question

I'm designing my own augmented reality application. I have already detected the 4 corners for the patterns I'm working with. After detecting the 4 corners in a proper order I'm passing them into cvFindExtrinsicCameraParams2. I'm getting nice results for the rotation and translation of the object respect to the camera frame. Now I have to put that information (Rotation Vector and Translation Vector) into OpenGL to draw something. Naturally, I'm using cvRodrigues2 to get the rotation matrix from rotation vector. In addition to this I'm watching the camera with QGlWidget in this way:

GLWidget.h

#ifndef _GLWIDGET_H
#define _GLWIDGET_H

#include <QtOpenGL/QGLWidget>
#include <cv.h>
#include <cxcore.h>

class GLWidget : public QGLWidget {

    Q_OBJECT // must include this if you use Qt signals/slots

public:
    GLWidget(QWidget *parent = NULL);
    IplImage *img;
    void setImage(IplImage *imagen);
protected:
    void initializeGL();
    void resizeGL(int w, int h);
    void paintGL();
};

#endif  /* _GLWIDGET_H */

GLWidget.cpp

#include <QtGui/QMouseEvent>
#include "GLWidget.h"

GLWidget::GLWidget(QWidget *parent) : QGLWidget(parent) {
    this->img = 0;
}

void GLWidget::initializeGL() {
    glDisable(GL_TEXTURE_2D);
    glDisable(GL_DEPTH_TEST);
    glDisable(GL_COLOR_MATERIAL);
    glEnable(GL_BLEND);
    glEnable(GL_POLYGON_SMOOTH);
    glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
    glClearColor(0, 0, 0, 0);
}

void GLWidget::resizeGL(int w, int h) {
    glViewport(0, 0, w, h);
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    gluOrtho2D(0, w, 0, h); // set origin to bottom left corner
//    gluPerspective(52.0f, 1.3333f, 0.1f, 100.0f);
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();

}

void GLWidget::paintGL() {

    if(this->img)
    {
        glClear (GL_COLOR_BUFFER_BIT);
        glClearColor(0.0,0.0,0.0,1.0);
        glMatrixMode(GL_PROJECTION);
        glPushMatrix();
        glLoadIdentity();
        gluOrtho2D(0.0,img->width, 0.0, img->width);
        glMatrixMode(GL_MODELVIEW);
        glPushMatrix();
        glLoadIdentity();
        glDrawPixels(img->width,img->height,GL_RGB,GL_UNSIGNED_BYTE,img->imageData);
        glMatrixMode(GL_PROJECTION);
        glPopMatrix();
        glMatrixMode(GL_MODELVIEW);
        glPopMatrix();
    }
}
void GLWidget::setImage(IplImage *imagen)
{
    this->img = imagen;
    this->updateGL();
}

Ok, so, as an additional note in the constructor of my MainWindow I have something like:

windowGL = new GLWidget();
windowGL.setParent(this);

In order to put the GL window inside the MainWindow. Also I created a new variable inside GLViewer.h called :

double modelViewMatrix[16] = {0.0};

So, in order to display a 3D object I created a method inside GLViewer like this:

void GlViewer::setModelView(double cameraMatrix[]){
    for(int i=0;i<16;i++){
    this->modelViewMatrix[i] = cameraMatrix[i];
    }
    this->updateGL();
}

After that, I'm putting into GLViewer::paintGL() method something like this after drawing the image with glDrawPixels() and obviously after popping matrices:

    glMatrixMode(GL_MODELVIEW);
    glLoadMatrixd(this->modelViewMatrix);
    glPushMatrix();
    glColor3f(1,0,0);

    glPushAttrib(GL_COLOR_BUFFER_BIT | GL_POLYGON_BIT | GL_ENABLE_BIT) ;
    glDisable(GL_LIGHTING) ;
    glPolygonMode(GL_FRONT_AND_BACK, GL_LINE) ;
    glLineWidth(3);

    glBegin(GL_LINES) ;
            glColor3f(1,0,0) ;
            glVertex3f(0,0,0) ;
            glVertex3f(100,0,0);

            glColor3f(0,1,0) ;
            glVertex3f(0,0,0) ;
            glVertex3f(0,100,0);

            glColor3f(0,0,1) ;
            glVertex3f(0,0,0) ;
            glVertex3f(0,0,100);
    glEnd() ;

    glLineWidth(1) ;
    glPopAttrib() ;
    glMatrixMode(GL_MODELVIEW);
    glPopMatrix();

So, for modelViewMatrix I'm taking into account that I have to order into columns instead of rows, or transpose, whatever you want... So, naturally, as the last step I'm calling the created function GLWidget::setModelView after I converted the camera intrinsic and extrinsic into an array:

windowGL.setModelView(convertedFromIntrExtr);

However I'm seeing nothing... I've tried a code and I get that working drawing the axes but without the projection matrix... However, when I use glutPerspective() (obviously after glMatrixMode(GL_PROJECTION)) I just can't see anything... so, I don't know if someone out there has a working code in Qt with augmented reality without using ARToolkit because, as I said, I'm getting the extrinsic camera parameters for my own... So, If anyone has a working code in which you get the extrinsic camera parameters and translate them into openGL projection and modelview matrices... hmm well, would be very helpful... this is because I found this example:

http://old.uvr.gist.ac.kr/wlee/web/techReports/ar/Camera%20Models.html

I followed all steps to achieve that conversion between the 2 camera models without success... I'll really appreciate if someone has a working code... thanks!!!

Answer 1

I've worked with CV for ego-pose estimation on The Artvertiser, and this stuff can do your head in.

First, you need to figure out why you can't see anything. It could be for a number of reasons, the most likely being an inversion of the camera look-at direction (handedness confusion), so the geometry is actually 'behind' the camera in OpenGL space. But the first debugging things I would try would be:

• draw a ground grid by looping from -1 to 1 at steps of 0.1f in x and z dimensions, drawing glLines to define a portion of the y plane (assuming righthanded coordinate system with y=up/down: x is right, y is up, z is pointing toward you out of the screen) (this way you can get a feel for the scale of the world and the orientation that your camera matrix is giving you)
• geometry 'behind' the camera (catch keypresses to move the camera backwards and forwards, and move around a bit and see if you can find the geometry)
• geometry scale too big/too small (catch keypresses to adjust global scale using glScalef right after setting up the camera)
• geometry outside of the clipping plane (adjust the clipping plane)

Also: this looks a bit suspect: this->modelViewMatrix[i] = cameraMatrix[i]; Have you tried inverting cameraMatrix first?