Qt QML Camera to C++ QImage on Android

Question

I have a Qt5.4-based program with some image processing. I use QCamera with my videoSurface (derived from QAbstractVideoSurface) to get VideoFrames. It works good on Windows.

But now I need Android version of my app. I found out that QCamera do not work on Android. But I see that QML Camera example run on Android with no problems.

So I decided to rewrite my application in QML. The main problem: I can't access QML Camera surface in C++.

void myVideoOutput::setSource(QObject *source)
{
    qDebug() << Q_FUNC_INFO << source;

    if (source == m_source.data())
        return;
    m_source = source;
    if (m_source) {
        const QMetaObject *metaObject = m_source.data()->metaObject();

        QStringList properties;
        for(int i = metaObject->propertyOffset(); i < metaObject >propertyCount(); ++i)
            properties << QString::fromLatin1(metaObject->property(i).name());
        qDebug() << properties;

    }
    .....
    emit sourceChanged();
}

This code give access to properties. But I can't access videoSurface this way (using QCamera I could do it). I wonder how QML Camera works? Is it based on QCamera? I see QCamera *m_camera in QDeclarativeCamera...

So I have 2 questions:

1. Is it possible to use QML Camera for postprocess images in C++? Working example would be very valuable.
2. Do you know other ways to capture video from Android camera in Qt?

Answer 1

1) Yes, it is possible. I´ve come around two ways to do it.

Using QAbstractVideoFilter alongside with QVideoFilterRunnable classes (QT 5.5 only! ) which are plain great. They were developed specifically for this case scenario, and are pretty easy to use.

There are a few good examples on the web using it:

https://blog.qt.io/blog/2015/03/20/introducing-video-filters-in-qt-multimedia/

http://code.qt.io/cgit/qt/qtmultimedia.git/tree/examples/multimedia/video/qmlvideofilter_opencl

The downside to this approach is, like was said here , is that on Android devices the QVideoFrame pointer doesnt has raw pixel data, instead, it has an OpenGL Texture which need to be read back (the second example I posted has a workaround solving this), thus making this approach not realy good for real time purposes IMHO.

What I ended up using to solve this problem was the QVideoProbe class.

First you have to name the instance of your QML camera:

    Camera {
    id: camera

    objectName: "qrCameraQML"
}

Then you get this instance from C++ side, something like:

QObject *qmlCamera = engine.rootObjects().at(0).findChild<QObject*>("qrCameraQML");

The QML camera instance actually has a QVariant element accessible only via C++ that can be cast into a QCamera* :

camera_ = qvariant_cast<QCamera*>(qmlCamera->property("mediaObject"));

Then all you have to do is to connect the probe to a slot that will actually handle the QVideoFrames and then set the source of the probe as the QCamera* previouslly cast:

    connect(&probe_,SIGNAL(videoFrameProbed(QVideoFrame)),this,SLOT(handleFrame(QVideoFrame)));

probe_.setSource(camera_);

On my example camera_ and probe_ are simply:

    QCamera *camera_;

QVideoProbe probe_;

This approach on my experience was a lot faster(for android platforms) than using the qt video filter classes, but it has the disadvantage that you basically only read the video output from qml, and AFAIK you wont be able to send postprocessed videoframes back to qml.

If you really need to send the processed images back to qml i'd advise you to try the first approach and see what happens.

2)Not with Qt AFAIK, maybe with OpenCv, or some other lib.

Answer 2

I like to highlight @waldez-junior first answer. In QML, you add your QAbstractVideoFilter component to the VideoOutput.

Camera {
    id: camera
}

VideoOutput {
    id: videoOutput
    source: camera
    filters: [ videoFilter ]
    autoOrientation: true
}

MyVideoFilter {
    id: videoFilter
    // orientation: videoOutput.orientation
}

In C++, you implement QAbstractVideoFilter component, here's a minimal example:

class MyVideoFilter : public QAbstractVideoFilter
{
    Q_OBJECT

public:
    QVideoFilterRunnable *createFilterRunnable() Q_DECL_OVERRIDE
    {
        return new CustomFilterRunnable(this);
    }
};

class MyVideoFilterRunnable : public QVideoFilterRunnable
{
public:
    QVideoFrame run(QVideoFrame* input, const QVideoSurfaceFormat& surfaceFormat, RunFlags flags)
    {
        if (!input->isValid())
        {
            return *input;
        }

        // do stuff with input
        return *input;
    }
};

```

There's an QAbstractVideoFilter example in Qt source code: http://code.qt.io/cgit/qt/qtmultimedia.git/tree/examples/multimedia/video/qmlvideofilter_opencl.

To make things easier, consider using the Qt internal function qt_imageFromVideoFrame for converting a QVideoFrame to a QImage. This code works for NoHandle case and works on most platforms. Unfortunately, it doesn't work for at lot of Android devices because QVideoFrame::map() will return false.

extern QImage qt_imageFromVideoFrame(const QVideoFrame& f);

For Android, you need to handle the GLTextureHandle case where you use OpenGL to populate a QImage.

On some devices, the image internal bit buffers will appear flipped.

#ifdef Q_OS_ANDROID
    bool flip = true;
#else
    bool flip = surfaceFormat.scanLineDirection() == QVideoSurfaceFormat::BottomToTop;
#endif

On some devices, the image may also be rotated. The best way for dealing with rotation is set autoOrientation: true in the VideoOutput component. Then, your component can just take a copy of videoOutput.orientation.

Knowing how an image is flipped and rotated will assist in video recognition applciations (e.g. face recognition).

I've also created a minimum working sample on https://github.com/stephenquan/MyVideoFilterApp

Answer 3

1. I think the answer above sufficiently explained processing from the QML Camera
2. Yes, there are other possibilities I found this project helped me a lot: https://github.com/rferrazz/CvCamView register the plugin to QML and it can be used like this:

import QtQuick 2.3
import QtQuick.Window 2.2
import CVComponents 1.0

Window {
    visible: true
    CVCAM{
        id: camera
        width: 640
        height: 480
        deviceId: "0"
        imageHeight: 640
        imageWidth: 480
    }

}

Processing the image is as very simple. The class only paints the item so it can be used on QML all other processing goes on in the back end, so the Mat image used by the camera can be used for processing.