OpenCV Android - color issue using CameraBridgeViewBase

Question

I'm encountering a strange problem using Android emulators & OpenCV CameraBridgeViewBase.

Using onCameraFrame I get a picture that looks like it wasn't decoded properly.

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    return inputFrame.rgba();
}

Using 'inputFrame.gray()' I get what's expected - black and white image without artifacts or any other issues.

That's what I get:

One more picture (a bigger one)

What I've tried so far:

1. Different API levels (from 15 up to 21).
2. Different emulators: Genymotion & Google Android emulator.
3. Different platform architectures - both ARM and Intel x86.
4. Launching emulator on different laptop with Linux: it works as expected, the issue is gone!
5. Launching apps, using OpenCV, downloaded from Play Store. They DO work! However:
   1. Launch app that works as expected, then close it.
   2. Launch your app (or one of the OpenCV tutorials), then close it.
   3. Launching app from the 5.1 again I see that it's affected by the same bug!
6. Different OpenCV versions (2.4.9 and 2.4.10).
7. Different versions of the OpenCV manager (one from the Play Store and 2.4.9 & 2.4.10 from OpenCV package).
8. Finally, as I noticed in 5.2, precompiled tutorial .apk files from the OpenCV package are affected by the issue too.

Everything works as expected on my real android devices.

After looking at sources of the CameraBridgeViewBase and Java/Native camera classes I came to the decision that the problem occurs while decoding image. Probably there is a problem with platform-specific camera output format (YUV, NV21). However, it's strange that .gray() gives out a proper image (without artifacts).

I'm using Mac OS X 10.10 Yosemite and MacBook Air with "Facetime HD" camera if that matters.

Any ideas on how to overcome this problem & help in finding the root of the problem are greatly appreciated!

Answer 1

So, after drilling into the problem I've found the root of the issue.

Let's take a look at the OpenCV JavaCameraView class and its CameraBridgeViewBase base class. The problem was that camera frames received as byte[] array in onPreviewFrame method were decoded incorrectly.

The exact place of code where the decoding process takes place is an implementation of the Mat rgba() method in inner JavaCameraFrame class of the JavaCameraView:

    public Mat rgba() {
        Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4);
        return mRgba;
    }

As we see, Imgproc.cvtColor(...) method is used in order to convert frame from YUV to RGBA. NV21 YUV -> RGBA conversion takes place there. During the initialization process we set the format to NV21, so this should be right. Moreover, every Android device should support NV21. Also, we can check whether device accepted the format using debugger:

protected boolean initializeCamera(int width, int height) {
    ...
    params.setPreviewFormat(ImageFormat.NV21);
    ...
    mCamera.setParameters(params);
    ...
    params = mCamera.getParameters();
    Log.d(TAG, String.format("Actual preview format is 0x%X", params.getPreviewFormat()));
}

Both phone (HTC Sensation) and emulator reported to be using NV21 indeed.

However, if we change COLOR_YUV2RGBA_NV21 to COLOR_YUV2RGB_I420 (YV12 and I420 is the same thing, just with Y and V inverted;) we'll see that emulator will get a proper color space finally. Changing NV21 to YV12 in params.setPreviewFormat(ImageFormat.NV21); we'll get similar results. Looks like there's bug either in Imgproc.cvtColor, or in Android.

Here comes the solution. Change public Mat rgba() the following way:

    public Mat rgba() {
        if (previewFormat == ImageFormat.NV21) {
            Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4);
        }
        else if (previewFormat == ImageFormat.YV12) {
            Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGB_I420, 4);  // COLOR_YUV2RGBA_YV12 produces inverted colors
        }
        return mRgba;
    }

previewFormat is a new int variable, it's declared this way:

private int previewFormat = ImageFormat.NV21;

Add following changes to the initialization:

protected boolean initializeCamera(int width, int height) {
        ...
                params.setPreviewFormat(ImageFormat.NV21);
                // "generic" = android emulator
                if (Build.BRAND.equalsIgnoreCase("generic")) {
                    params.setPreviewFormat(ImageFormat.YV12);
                }
                ...
                mCamera.setParameters(params);
                params = mCamera.getParameters();
                previewFormat = params.getPreviewFormat();
        ...
}

Important:
Please note: this is just a temporary solution in order to make OpenCV usable with emulator in my case. Further research should be done. It's quite easy to check whether device uses correct image format in onPreviewFrame. I'll get back to this when I have some time.