From android.media.Image to Mat

Question

I am using Android Camera2 API to acquire frames from the camera. I can get an ImageReader that reads Image objects with the acquireLatestImage() method.

Since I need to process the acquired frames, I have to convert each Image object into a Mat object. I suppose this is a quite common problem, so I expected to find a convenient OpenCV method to do that. However I didn't find any.

Do you have an idea on how to get Mat objects from the camera frames with Android Camera2? Thanks

Answer 1

This depends on what ImageFormat you set for your ImageReader class. This code below assumes that it's using the JPEG format.

Image image = reader.acquireLatestImage();

Mat buf = new Mat(image.getHeight(), image.getWidth(), CvType.CV_8UC1);

ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);
buf.put(0, 0, bytes);

// Do note that Highgui has been replaced by Imgcodecs for OpenCV 3.0 and above
Mat mat = Highgui.imdecode(buf, IMREAD_COLOR);

image.close();

Do note that imdecode is quite slow. I'm not entirely sure but I think YUV_420_888 would be faster to convert to a Mat object, but I haven't figured out how to do it myself. I saw this post attempt the YUV conversion to a Mat,