Most Efficient way to send Image in YUV_420_888 from Android to JNI in OpenCV Mat

Question

I have an android application where I want to run some OpenCV image processing on live images from android Camera2 API. Currently, If I don't do any processing, I am able to receive images in OnImageAvailble function at 30 fps for a regular 1280x720 frame.

Now as a dirty hack I am requesting Images from ImageReader in JPEG format and then passing the Bitmap to jni which brings down the performance a lot.

What's the most efficient way to pass the YUV frame to jni in cv Mat object. Also, I want to convert this frame to RGB for further processing. Should I change the format on Java side or should I pass the Mat object to jni and convert the colorspace there only.

Answer 1

Anything you do in C++ is much faster than the Java equivalent for obvious reasons, including YUV to RGB transformations (even if the Java implementation relies on compiled libraries).

You can directly pass a pointer from your existing Mat in java directly to C++ through JNI. Supposing I want to do Canny() using C++ and JNI, and I have a JNI function defined like this:

// In Java
public static native boolean nativeCanny(long iAddr);

Notice the long iAddr parameter, that's a direct pointer to my Mat in Java. You invoke it like this:

// In Java
nativeCanny(myImage.getNativeObjAddr());

The implementation of this function in C++ would receive this pointer in a similar way to this (replace long with jlong if this doesn't work):

// In C++
JNIEXPORT jboolean JNICALL
VeryLongName_nativeCanny(JNIEnv *env, jobject instance, long iAddr) {
    cv::Mat* img = (cv::Mat*) iAddr;
    cv::Canny(*img, *img, 80, 100, 3);
    return true;
}

And whatever I did to the img Mat, happens in the java myImage Mat as well, after all it's a pointer so we never made a copy.

As far as I know, that's as fast as it gets.

Answer 2

Maybe helpful to you: Since we do a lot of image processing we wrote a library for that purpose in our company. It's not written in C but it's quite performant. After the conversion, you can simply pass the Mat pointer down to your C code via JNI.

It converts YUVs (Standard Android Camera Format YUV_420_888) to RGB Mats. In addition, it also allows efficient clipping of the YUV before the conversion (important for very big images, otherwise you would need to convert the whole image, and clip afterward which is expensive). Usage is very simple:

Mat mat = Yuv.toMat(image)

https://github.com/quickbirdstudios/yuvToMat