Integrating OpenCV with larger programs

Question

Can anyone recommend a how-to guide or provide a brief overview of what's involved with integrating OpenCV with larger GUI-based programs? What are the popular ways to do it?

Particularly, processing video with OpenCV while doing video capture/preview without using HighGUI seems especially arcane. I hope someone can demystify this.

My particular configuration is with either Juce or Qt depending on what can be done. The cross platform thing is not critical -- if there is an awesome way of doing this in Windows, I might be convinced. The availability of community support is important.

I have heard that HighGUI is entirely for testing and unsuitable for real applications. Someone recommended the VideoInput library, but it is experimental.

---

Key points from answers:

• Use Qt (because Qt is great and has a big community).
• Open a new thread to run cv::VideoCapture in a loop and emit signal after frame capture. Use Qt's msleep mechanism, not OpenCV. So, we are still using OpenCV highgui for capture.

• Convert cv::Mat to QtImage:

  QImage qtFrame(cvFrame.data, cvFrame.size().width, cvFrame.size().height, cvFrame.step, QImage::Format_RGB888);

  qtFrame = qtFrame.rgbSwapped();

• Optional: Render with GLWidget. Convert QtImage to GLFormat with Qt built-in method:

  m_GLFrame = QGLWidget::convertToGLFormat(frame);

  this->updateGL();

Answer 1

Here is how I am doing it with Qt. You are welcome to use whatever may be useful to you :)

/// OpenCV_GLWidget.h
#ifndef OPENCV_GLWIDGET_H_
#define OPENCV_GLWIDGET_H_

#include <qgl.h>
#include <QImage>

class OpenCV_GLWidget: public QGLWidget {
public:
    OpenCV_GLWidget(QWidget * parent = 0, const QGLWidget * shareWidget = 0, Qt::WindowFlags f = 0);
    virtual ~OpenCV_GLWidget();

    void renderImage(const QImage& frame);
protected:
    virtual void paintGL();
    virtual void resizeGL(int width, int height);

private:
    QImage m_GLFrame;
};

#endif /* OPENCV_GLWIDGET_H_ */

/// OpenCV_GLWidget.cpp
#include "OpenCV_GLWidget.h"

OpenCV_GLWidget::OpenCV_GLWidget(QWidget* parent, const QGLWidget* shareWidget, Qt::WindowFlags f) :
QGLWidget(parent, shareWidget, f)
{
    // TODO Auto-generated constructor stub

}

OpenCV_GLWidget::~OpenCV_GLWidget() {
    // TODO Auto-generated destructor stub
}

void OpenCV_GLWidget::renderImage(const QImage& frame)
{
    m_GLFrame = QGLWidget::convertToGLFormat(frame);
    this->updateGL();
}

void OpenCV_GLWidget::resizeGL(int width, int height)
{
    // Setup our viewport to be the entire size of the window
    glViewport(0, 0, width, height);

    // Change to the projection matrix and set orthogonal projection
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glOrtho(0, width, height, 0, 0, 1);
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
}

void OpenCV_GLWidget::paintGL() {
    glClear (GL_COLOR_BUFFER_BIT);
    glClearColor (0.0, 0.0, 0.0, 1.0);
    if (!m_GLFrame.isNull()) {
        m_GLFrame = m_GLFrame.scaled(this->size(), Qt::IgnoreAspectRatio, Qt::SmoothTransformation);

        glEnable(GL_TEXTURE_2D);
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
        glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, m_GLFrame.width(), m_GLFrame.height(), 0, GL_RGBA, GL_UNSIGNED_BYTE, m_GLFrame.bits() );
        glBegin(GL_QUADS);
        glTexCoord2f(0, 0); glVertex2f(0, m_GLFrame.height());
        glTexCoord2f(0, 1); glVertex2f(0, 0);
        glTexCoord2f(1, 1); glVertex2f(m_GLFrame.width(), 0);
        glTexCoord2f(1, 0); glVertex2f(m_GLFrame.width(), m_GLFrame.height());
        glEnd();
        glDisable(GL_TEXTURE_2D);

        glFlush();
    }
}

This class handles the rendering of the image onto a promoted QWidget. Next, I created a thread to feed the widget. (I cheated using the Qt signal-slot architecture here because it was easy...may not be the best performer in the book, but it should get you started).

void VideoThread::run()
{
    cv::VideoCapture video(0);

    while(!m_AbortCapture)
    {
        cv::Mat cvFrame;
        video >> cvFrame;

        cv::Mat gray(cvFrame.size(), CV_8UC1);
        cv::GaussianBlur(cvFrame, cvFrame, cv::Size(5, 5), 9.0, 3.0, cv::BORDER_REPLICATE);
        cv::cvtColor(cvFrame, gray, CV_RGB2GRAY);

        m_ThresholdLock.lock();
        double localThreshold = m_Threshold;
        m_ThresholdLock.unlock();

        if(localThreshold > 0.0)
        {
            qDebug() << "Threshold = " << localThreshold;
            cv::threshold(gray, gray, localThreshold, 255.0,  cv::THRESH_BINARY);
        }

        cv::cvtColor(gray, cvFrame, CV_GRAY2BGR);

        // convert the Mat to a QImage
        QImage qtFrame(cvFrame.data, cvFrame.size().width, cvFrame.size().height, cvFrame.step, QImage::Format_RGB888);
        qtFrame = qtFrame.rgbSwapped();

        // queue the image to the gui
        emit sendImage(qtFrame);
        msleep(20);
    }
}

Took me a bit to figure that out, so hopefully it will help you and others save some time :D

Answer 2

Create an openCV image to hold the image you have captured.

Do processing on it and then copy the data into the image you want to display (eg QImage)

You can optomise things by creating the opencv cv::Mat image to share the memory with the

QImage but since QImage generally uses ARGB and most image processing tasks are better done as greyscale or RGB it's probably better to copy images and convert between them using the opencv cvtColor() function

Then simply include include the opencv headers, and link with the opencv libs - there are guides on the opencv wiki for your particular environment