Efficient integration of Qt and OpenCV

Question

I am working on an interactive application which needs to read and manipulate several very large images at once (25 images at a time, roughly 350 Mb total size). OpenCV is quite speedy and handles the algorithms with relative ease. But drawing them with Qt is proving to be a problem. Here are two less-than-ideal solutions I have tried.

Solution 1 (too slow)

Every time you need to draw a different OpenCV image, convert it to a QImage and draw that. The conversion, unfortunately, takes a while and we cannot switch between images at interactive speeds.

Solution 2 (too memory-intensive)

Maintain two stacks of images, one for OpenCV and one for Qt. Use the appropriate one at the appropriate time.

I have direct access to the OpenCV pixel data. I know the width and height of the image, and I know that pixels are 3-byte RGB values. It seems like it should be possible to draw the OpenCV image quickly without copying it to a QImage container that (as far as I can tell) just contains a duplicate of the data.

Where do I need to look to get this kind of capability out of Qt?

Answer 1

I don't know if this might be useful to you now after 3 months. But I am having the same kind of application where I have to manipulate a stream of images using OpenCV and display it on a QT interface. After googling around quite a bit, I came across a very slick solution. Use opengl's glDrawPixels to draw raw image data directly on the Qt interface. Best part, u don't have to write any extra conversion code. Just the basic code for opengl for setting up a viewport and coordinate. Check out the code which has a function which takes an IplImage* pointer and uses that data to draw the image. You might need to tweak the parameters(especially the WIDTH and HEIGHT variables) a bit to display an image with a specific size. And yeah, I don't know what build system you are using. I used cmake and had to setup dependencies for opengl although I am using Qt's opengl libraries.

I have implemented a class QIplImage which derives from QGLWidget and overriden its paintGL method to draw the pixel data on to the frame.

//File qiplimage.h
class QIplImage : public QGLWidget
{
  Q_OBJECT

 public:
    QIplImage(QWidget *parent = 0,char *name=0);
   ~QIplImage();
   void paintGL();
   void initializeGL();
   void resizeGL(int,int);
   bool drawing;

 public slots:
   void setImage(IplImage);

 private:
  Ui::QIplImage ui;
  IplImage* original;
  GLenum format;
  GLuint texture;
  QColor bgColor;
  char* name;
  bool hidden;
  int startX,startY,endX,endY;
  QList<QPointF*> slopes;
  QWidget* parent;
  int mouseX,mouseY;

};
//End of file qiplimage.h

//file qiplimage.cpp
#include "qiplimage.h"
#include <Globals.h>

QIplImage::QIplImage(QWidget *parent) :
    QGLWidget(parent)
{

}
QIplImage::QIplImage(QWidget *parent,char* name): QGLWidget(parent)
{
     ui.setupUi(this);
    //This is required if you need to transmit IplImage over
    // signals and slots.(That's what I am doing in my application
    qRegisterMetaType<IplImage>("IplImage");
    resize(384,288);
    this->name=name;
    this->parent=parent;
    hidden=false;
    bgColor= QColor::fromRgb(0xe0,0xdf,0xe0);

    original=cvCreateImage(cvSize(this->width(),this->height()),IPL_DEPTH_8U,3);
    cvZero(original);
    switch(original->nChannels) {
        case 1:
            format = GL_LUMINANCE;
            break;
        case 2:
            format = GL_LUMINANCE_ALPHA;
            break;
        case 3:
            format = GL_BGR;
            break;
        default:
            return;
}
    drawing=false;
    setMouseTracking(true);
    mouseX=0;mouseY=0;
    initializeGL();

}
void QIplImage::initializeGL()
{
   qglClearColor(bgColor);  
   //glClearColor(0.5f, 0.5f, 0.5f, 1.0f);              
   glDisable(GL_DEPTH_TEST);
   glMatrixMode(GL_PROJECTION);
   glLoadIdentity();
       glOrtho(0,this->width(),this->height(),0.0f,0.0f,1.0f);
   glMatrixMode(GL_MODELVIEW);
   glLoadIdentity();

   glEnable(GL_TEXTURE_2D);
   glGenTextures(3,&texture);
   glBindTexture(GL_TEXTURE_2D,texture);
   glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_NEAREST);
   glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_NEAREST);
   glBindTexture(GL_TEXTURE_2D,texture);                glTexImage2D(GL_TEXTURE_2D,0,GL_RGB,this->width(),this->height(),0,GL_BGR,GL_UNSIGNED_BYTE,NULL);
   glDisable(GL_TEXTURE_2D);


}
void QIplImage::setImage(IplImage image){
original=&image;
//cvShowImage(name,original);

updateGL();
}

void QIplImage::paintGL (){
   glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glDisable(GL_DEPTH_TEST);
if(!hidden){
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
            glOrtho(0.0f,this->width(),this->height(),0.0f,0.0f,1.0f);
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
    glEnable(GL_TEXTURE_2D);
            glBindTexture(GL_TEXTURE_2D,texture);
            glTexImage2D(GL_TEXTURE_2D,0,GL_RGB,original->width,original->height,0,GL_BGR_EXT,GL_UNSIGNED_BYTE,original->imageData);
    glBegin(GL_QUADS);
            glTexCoord2i(0,1); glVertex2i(0,this->height());
    glTexCoord2i(0,0); glVertex2i(0,0);
            glTexCoord2i(1,0); glVertex2i(this->width(),0);
            glTexCoord2i(1,1); glVertex2i(this->width(),this->height());
    glEnd();
    glFlush();
    }

}


void QIplImage::resizeGL(int width,int height){

    glViewport(0,0,this->width(),this->height());
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();       
    glOrtho(0.0f,this->width(),this->height(),0.0f,0.0f,1.0f);
    glMatrixMode(GL_MODELVIEW);         
    glLoadIdentity();
 }

Hope that helps.

Answer 2

You can share the data between QImage and openCV - both of them have a ctor which uses existing data - supplied by a pointer.

cv::Mat(int _rows, int _cols, int _type, void* _data, size_t _step=AUTO_STEP)
QImage ( uchar * data, int width, int height, int bytesPerLine, Format format)

There might be an issue with the padding if the rows don't end up being multiples of 4bytes but I would expect the padding to align on both types with the same pixel size - at least on the same hardware

One issue is that openCV uses BGR by default which isn't very optimal for QImage (or any other display). Although I'm not sure that QImage::Format_ARGB32_Premultiplied is necessarily that much quicker anymore on Qt which use accelerated openGL for rendering QImage.

An alternative is to use opencv then copy the resulting data direct to an openGL texture and then use QGlWidget to display the image without another copy.