Is it possible to use Camera2 with Google Vision API

Question

Is it possible to detect faces using Camera2 with Google Vision API only ? I could not find a way to integrate it.

Answer 1

Yes, It is possible to use Camera2 API with Google Vision API.

To start with, the Google Vision API Face Detector receives a Frame object that uses to analyze (detect faces and its landmarks).

The Camera1 API provides the preview frames in NV21 image format, which is ideal for us. The Google Vision Frame.Builder supports both setImageData (ByteBuffer in NV16, NV21 or YV12 image format) and setBitmap to use a Bitmap as the Preview Frames to process.

Your issue is that the Camera2 API provides the preview frames in a different format. It is YUV_420_888. To make everything work you have to convert the preview frames into one of the supported formats.

Once you get the Camera2 Preview Frames from your ImageReader as Image you can use this function to convert it to a supported format (NV21 in this case).

private byte[] convertYUV420888ToNV21(Image imgYUV420) {
    // Converting YUV_420_888 data to YUV_420_SP (NV21).
    byte[] data;
    ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer();
    ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer();
    int buffer0_size = buffer0.remaining();
    int buffer2_size = buffer2.remaining();
    data = new byte[buffer0_size + buffer2_size];
    buffer0.get(data, 0, buffer0_size);
    buffer2.get(data, buffer0_size, buffer2_size);
    return data;
}

Then you can use the returned byte[] to create a Google Vision Frame:

outputFrame = new Frame.Builder()
    .setImageData(nv21bytes, mPreviewSize.getWidth(), mPreviewSize.getHeight(), ImageFormat.NV21)
    .setId(mPendingFrameId)
    .setTimestampMillis(mPendingTimeMillis)
    .setRotation(mSensorOrientation)
    .build();

Finally, you call the detector with the created Frame:

mDetector.receiveFrame(outputFrame);

Anyway, if you want to know more about this you can test my working example available for free on GitHub: Camera2Vision. I hope I've helped :)

Answer 2

Ok, I found this camera2 with mobile vision? #65

There are no near term plans for a camera2 version of the CameraSource class in the official API. However, given how the API is structured, an alternate version of CameraSource could be written by the developer community that uses camera2. All of the existing APIs for working with frames and detectors are sufficient to support a camera2 implementation as well.