QT GUI freezes when capturing video from webcam using OpenCV

Question

I am using Opencv for some real-time video processing.

As a front-end I am using QT framework.

On my GUI, I have an input image window (mapped to a Label) and an output image window (mapped to another Label) and 3 push buttons. One to Start input video capture, the second to process the video (code not written yet), and third to Exit.

I am currently able to stream my video and display it on the Front-end. But this locks my GUI and am unable to Exit.

I tried using QTimers (using suggestions from this and the QT forum), but my GUI still remains locked.

Would appreciate if someone can point my in the right direction.

Below is the code:

mainwindow.h

#ifndef MAINWINDOW_H
#define MAINWINDOW_H

#include <QMainWindow>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>   // for cvtColor
#include <iostream>
#include <QTimer>

namespace Ui {
class MainWindow;
}

class MainWindow : public QMainWindow
{
    Q_OBJECT

public:
    explicit MainWindow(QWidget *parent = 0);
    ~MainWindow();

private slots:
    void on_buttonCaptureVideo_clicked();

    void on_buttonExit_clicked();

public slots:
    virtual void doNextFrame() {repaint();}

private:
    Ui::MainWindow *ui;
    CvCapture *capture;          // OpenCV Video Capture Variable
    IplImage *frame;            // Variable to capture a frame of the input video
    cv::Mat source_image;     // Variable pointing to the same input frame
    cv::Mat dest_image;      // Variable to output a frame of the processed video
    QTimer *imageTimer;
};

#endif // MAINWINDOW_H

mainwindow.cpp

#include "mainwindow.h"
#include "ui_mainwindow.h"

MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent),
    ui(new Ui::MainWindow)
{
    ui->setupUi(this);
}

MainWindow::~MainWindow()
{
    delete ui;
    cvReleaseImage(&frame);
    cvReleaseCapture(&capture);
}

void MainWindow::on_buttonCaptureVideo_clicked()
{
    // Set to 25 frames per second

    const int imagePeriod = 1000/25;   // ms

    imageTimer = new QTimer(this);

    imageTimer->setInterval(imagePeriod);

    connect(imageTimer, SIGNAL(timeout()), this, SLOT(doNextFrame()));

    // Use the default camera
    capture = cvCreateCameraCapture(-1);

    while(capture)
    {
    // Capture a frame
    frame = cvQueryFrame(capture);

    // Point to the same frame
    source_image = frame;

    // Resize Image
    cv::resize(source_image, source_image, cv::Size(128,128) , 0, 0);

    // Change to RGB format
    cv::cvtColor(source_image,source_image,CV_BGR2RGB);

    // Convert to QImage
    QImage qimg = QImage((const unsigned char*) source_image.data, source_image.cols, source_image.rows, QImage::Format_RGB888); // convert to QImage

    // Display on Input Label
    ui->labelInputVideo->setPixmap(QPixmap::fromImage(qimg));

    // Resize the label to fit the image
    ui->labelInputVideo->resize(ui->labelInputVideo->pixmap()->size());

    }
}

void MainWindow::on_buttonExit_clicked()
{

    connect(ui->buttonExit, SIGNAL(clicked()), qApp, SLOT(closeAllWindows()));
}

Answer 1

When you click your button, the

while(capture) { ... }

loop will run forever, as capture will never be set to NULL. This means the code flow never leaves your loop and thus the main thread cannot process anything else, like e.g. repainting.

The QTimer will emit its timeout() signals, but they will be placed as Events in Qt's Event Queue. As long as your on_buttonCaptureVideo_clicked() method is running, those Events will not be processed.

Here are my suggestions how to make it work:

---

This code:

// Set to 25 frames per second  
const int imagePeriod = 1000/25;   // ms        
imageTimer = new QTimer(this);        
imageTimer->setInterval(imagePeriod);        
connect(imageTimer, SIGNAL(timeout()), this, SLOT(doNextFrame()));   
// Use the default camera            
capture = cvCreateCameraCapture(-1);  

belongs into the constructor of MainWindow as you want to set that up one time only. There is no need to do it again when the user clicks the button the second, third, etc time.

The code which is within your while loop should go into the doNextFrame() slot (without the while construct).

Then your button will only do

imageTimer->start();

and then e.g. do

imageTimer->stop();

when it is clicked again.

Example code:

void MainWindow::on_buttonCaptureVideo_clicked()
{
    if( imageTimer->isActive() )
    {
        imageTimer->stop();
    }
    else
    {
        imageTimer->start();
    }
}

---

What will happen if you do that?

When you click the button, your on_buttonCaptureVideo_clicked() clicked slot will be called from the GUI thread, the timer will be started, and the method will return almost instantly.
Now the GUI thread is free and able to handle repaints etc.
From that point on, the timer will be sending timeout() signals every 40ms. Whenever the GUI thread is free, it will handle this signal and call your doNextFrame slot.
This slot will capture the next frame and return when it is done. When it is done, the GUI thread will be able to process other Events (e.g. repaint) again.
As soon as you click the button again, the timer will stop, and no new timeout() events will be sent. If you still see a couple of frames after the button has been clicked, this could mean that the timer events were sent faster than they could be processed.

Answer 2

Preface: I am not strong in C++ so I cannot offer specific code, but I am experienced in PyQt

This is a common pitfall for newcomers to Qt. What it seems your on_buttonCaptureVideo_clicked is doing is entering into a loop in your main GUI thread to do work. In QT, you want to avoid doing anything busy in your main thread. The Qt eventloop needs to be able to constantly process and flush your GUI events as they come in. What you are doing is blocking the event loop.

There are two different things you can do here. The first is the more basic approach but allows you to see more immediate results. You can "pump" the eventloop. Depending on how fast your while loop iterates, you can call qApp->processEvents();. This will allow Qt to process the pending GUI events and make your app appear more responsive. Its basically sharing time between your while loop and the main loop. Maybe you want to call this on every nth frame. Depends how often you want to make sure the GUI refreshes.

The other option, which is more preferable, is to place your capture loop into a QThread. When a new frame is available you can emit a signal with the frame data. The signal will get placed into the Qt event loop to be processed with everything else and will not hold up your GUI. Generally this is the approach you want to take with any heavy crunching or long running callables.

Edit

I just realized that you start a QTimer in addition to doing a loop in the main thread. If you want to use a QTimer, and your image processing is not too heavy (it doesn't take long per cycle) then you should move everything into the doNextFrame and completely remove the while loop. If your doNextFrame is a heavy process, then you should be using a QThread and signals.