OpenCV VideoCapture removes alpha channel from video

Question

I have video with alpha channel and I am trying to place it over another video as follows:

public static void overlayImage(Mat background, Mat foreground, Mat output, Point location) {
        background.copyTo(output);

        for (int y = (int) Math.max(location.y, 0); y < background.rows(); ++y) {

            int fY = (int) (y - location.y);

            if (fY >= foreground.rows()) {
                break;
            }

            for (int x = (int) Math.max(location.x, 0); x < background.cols(); ++x) {
                int fX = (int) (x - location.x);
                if (fX >= foreground.cols()) {
                    break;
                }

                double opacity;
                double[] finalPixelValue = new double[4];

                opacity = foreground.get(fY, fX)[3];

                finalPixelValue[0] = background.get(y, x)[0];
                finalPixelValue[1] = background.get(y, x)[1];
                finalPixelValue[2] = background.get(y, x)[2];
                finalPixelValue[3] = background.get(y, x)[3];

                for (int c = 0; c < output.channels(); ++c) {
                    if (opacity > 0) {
                        double foregroundPx = foreground.get(fY, fX)[c];
                        double backgroundPx = background.get(y, x)[c];

                        float fOpacity = (float) (opacity / 255);
                        finalPixelValue[c] = ((backgroundPx * (1.0 - fOpacity)) + (foregroundPx * fOpacity));
                        if (c == 3) {
                            finalPixelValue[c] = foreground.get(fY, fX)[3];
                        }
                    }
                }
                output.put(y, x, finalPixelValue);
            }
        }
  }

When I run this function I get Nullpointer exception since apparently foreground Mat which is taken from VideoCapture like this:

capture.grab() && capture.retrieve(foregroundMat, -1);

retrieves only rgb image and removes alpha channel. The video file originally is perfectly fine, and it retrieved mat should be in rgba format but it is not. What might be a reason to this issue?

Answer 1

Unfortunately, OpenCV does not supports fetching video frames with Alpha channel. This is evident from this code snippet. The authors have conveniently assumed that a video file would always have RGB frames.

A quick hack could be replacing AV_PIX_FMT_BGR24 with AV_PIX_FMT_BGRA at the relevant places(2-3 instances) and re-building the library to get your code working. But this dirty hack would always generate RGBA frames for all video formats.

I am personally planning to create a PR with this fix, but it may take some time.

Other possible solution would be to use some other third party library to fetch the frames of .webm or .mov format and then process it using OpenCV.