convert QVideoFrame to QImage

Question

I want to get every frames from a QMediaPlayer and convert it to QImage (or cv::Mat)

so I used videoFrameProbed signal from QVideoProbe:

connect(&video_probe_, &QVideoProbe::videoFrameProbed, 
         [this](const QVideoFrame& currentFrame){
   //QImage img = ??
}

But I didn't find any way for getting QImage from QVideoFrame!

How can I convert QVideoFrame to QImage ?!

Answer 1

You can use QImage's constructor:

 QImage img( currentFrame.bits(),
             currentFrame.width(),
             currentFrame.height(),
             currentFrame.bytesPerLine(),
             imageFormat);

Where you can get imageFormat from pixelFormat of the QVideoFrame:

 QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(currentFrame.pixelFormat());

Answer 2

For QCamera output, that method doesn't always work. In particular, QVideoFrame::imageFormatFromPixelFormat() returns QImage::Format_Invalid when given QVideoFrame::Format_Jpeg, which is what's coming out of my QCamera. But this works:

QImage Camera::imageFromVideoFrame(const QVideoFrame& buffer) const
{
    QImage img;
    QVideoFrame frame(buffer);  // make a copy we can call map (non-const) on
    frame.map(QAbstractVideoBuffer::ReadOnly);
    QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(
                frame.pixelFormat());
    // BUT the frame.pixelFormat() is QVideoFrame::Format_Jpeg, and this is
    // mapped to QImage::Format_Invalid by
    // QVideoFrame::imageFormatFromPixelFormat
    if (imageFormat != QImage::Format_Invalid) {
        img = QImage(frame.bits(),
                     frame.width(),
                     frame.height(),
                     // frame.bytesPerLine(),
                     imageFormat);
    } else {
        // e.g. JPEG
        int nbytes = frame.mappedBytes();
        img = QImage::fromData(frame.bits(), nbytes);
    }
    frame.unmap();
    return img;
}

Answer 3

Qt has a private function qt_imageFromVideoFrame for converting QVideoFrame to QImage. If you want to use it, you'll need to:

1. Add QT += multimedia multimedia-private to your .pro file
2. Add #include "private/qvideoframe_p.h" to your .cpp file

3. Then you can use qt_imageFromVideoFrame with the following signature:

   QImage qt_imageFromVideoFrame( const QVideoFrame& f );

Be aware of the warning in the header:

//
//  W A R N I N G
//  -------------
//
// This file is not part of the Qt API.  It exists purely as an
// implementation detail.  This header file may change from version to
// version without notice, or even be removed.
//
// We mean it.
//

The implementation uses QVideoFrame::map() and does low-level manipulation with the bits. The converters handle a variety of conditions including YUV.

The qt_imageFromVideoFrame works on almost all platforms. i.e. Windows, macOS, Linux and iOS. The only platform it doesn't work reliably well is Android. There, you'll need to use OpenGL calls to extract the VideoFrame. Also, on Android, you need to call QImage:rgbSwapped() at the end because QImage stores images as ARGB whereas OpenGL retrieves them as ABGR.

QImage QVideoFrameToQImage( const QVideoFrame& videoFrame )
{
    if ( videoFrame.handleType() == QAbstractVideoBuffer::NoHandle )
    {
        QImage image = qt_imageFromVideoFrame( videoFrame );
        if ( image.isNull() ) return QImage();
        if ( image.format() != QImage::Format_ARGB32 ) return image.convertToFormat( QImage::Format_ARGB32 );
        return image;
    }
    if ( videoFrame.handleType() == QAbstractVideoBuffer::GLTextureHandle )
    {
        QImage image( videoFrame.width(), videoFrame.height(), QImage::Format_ARGB32 );
        GLuint textureId = static_cast<GLuint>( videoFrame.handle().toInt() );
        QOpenGLContext* ctx = QOpenGLContext::currentContext();
        QOpenGLFunctions* f = ctx->functions();
        GLuint fbo;
        f->glGenFramebuffers( 1, &fbo );
        GLint prevFbo;
        f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo );
        f->glBindFramebuffer( GL_FRAMEBUFFER, fbo );
        f->glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,  GL_TEXTURE_2D, textureId, 0 );
        f->glReadPixels( 0, 0,  videoFrame.width(),  videoFrame.height(), GL_RGBA, GL_UNSIGNED_BYTE, image.bits() );
        f->glBindFramebuffer( GL_FRAMEBUFFER, static_cast<GLuint>( prevFbo ) );
        return image.rgbSwapped();
    }
    return QImage();
}

I have a complete working app on GitHub that includes an implementation of the above function:

https://github.com/stephenquan/MyVideoFilterApp/blob/master/QVideoFrameToQImage.cpp