Is there a way to crop Image/ImageProxy (before passing to MLKit's analyzer)?

Question

I'm using CameraX's Analyzer use case with the MLKit's BarcodeScanner. I would like to crop portion of the image received from the camera, before passing it to the scanner.

What I'm doing right now is I convert ImageProxy (that I recieve in the Analyzer) to a Bitmap, crop it and then pass it to the BarcodeScanner. The downside is that it's not a very fast and efficient process.

I've also noticed the warning I get in the Logcat when running this code:

ML Kit has detected that you seem to pass camera frames to the detector as a Bitmap object. This is inefficient. Please use YUV_420_888 format for camera2 API or NV21 format for (legacy) camera API and directly pass down the byte array to ML Kit.

It would be nice to not to do ImageProxy conversion, but how do I crop the rectangle I want to analyze?

What I've already tried is to set a cropRect field of the Image (imageProxy.image.cropRect) class, but it doesn't seem to affect the end result.

Answer 1

Yes, it's true that if you use ViewPort and set viewport to yours UseCases(imageCapture or imageAnalysis as here https://developer.android.com/training/camerax/configuration) you can get only information about crop rectangle especially if you use ImageAnalysis(because if you use imageCapture, for on-disk the image is cropped before saving and it doesn't work for ImageAnalysis and if you use imageCapture without saving on disk) and here solution how I solved this problem:

1. First of all set view port for use cases as here: https://developer.android.com/training/camerax/configuration

2. Get cropped bitmap to analyze

   override fun analyze(imageProxy: ImageProxy) {
        val mediaImage = imageProxy.image
        if (mediaImage != null && mediaImage.format == ImageFormat.YUV_420_888) {
            croppedBitmap(mediaImage, imageProxy.cropRect).let { bitmap ->
                requestDetectInImage(InputImage.fromBitmap(bitmap, rotation))
                    .addOnCompleteListener { imageProxy.close() }
            }
        } else {
            imageProxy.close()
        }
    }
   
    private fun croppedBitmap(mediaImage: Image, cropRect: Rect): Bitmap {
        val yBuffer = mediaImage.planes[0].buffer // Y
        val vuBuffer = mediaImage.planes[2].buffer // VU
   
        val ySize = yBuffer.remaining()
        val vuSize = vuBuffer.remaining()
   
        val nv21 = ByteArray(ySize + vuSize)
   
        yBuffer.get(nv21, 0, ySize)
        vuBuffer.get(nv21, ySize, vuSize)
   
        val yuvImage = YuvImage(nv21, ImageFormat.NV21, mediaImage.width, mediaImage.height, null)
        val outputStream = ByteArrayOutputStream()
        yuvImage.compressToJpeg(cropRect, 100, outputStream)
        val imageBytes = outputStream.toByteArray()
   
        return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
    }
   

Possibly there is a loss in conversion speed, but on my devices I did not notice the difference. I set 100 quality in method compressToJpeg, but mb if set less quality it can improve speed, it need test.

upd: May 02 '21 :

I found another way without convert to jpeg and then to bitmap. This should be a faster way.

1. Set viewport as previous.

2. Convert YUV_420_888 to NV21, then crop and analyze.

    override fun analyze(imageProxy: ImageProxy) {
        val mediaImage = imageProxy.image
        if (mediaImage != null && mediaImage.format == ImageFormat.YUV_420_888) {
            croppedNV21(mediaImage, imageProxy.cropRect).let { byteArray ->
                requestDetectInImage(
                    InputImage.fromByteArray(
                        byteArray,
                        imageProxy.cropRect.width(),
                        imageProxy.cropRect.height(),
                        rotation,
                        IMAGE_FORMAT_NV21,
                    )
                )
                    .addOnCompleteListener { imageProxy.close() }
            }
        } else {
            imageProxy.close()
        }
    }
   
    private fun croppedNV21(mediaImage: Image, cropRect: Rect): ByteArray {
        val yBuffer = mediaImage.planes[0].buffer // Y
        val vuBuffer = mediaImage.planes[2].buffer // VU
   
        val ySize = yBuffer.remaining()
        val vuSize = vuBuffer.remaining()
   
        val nv21 = ByteArray(ySize + vuSize)
   
        yBuffer.get(nv21, 0, ySize)
        vuBuffer.get(nv21, ySize, vuSize)
   
        return cropByteArray(nv21, mediaImage.width, cropRect)
    }
   
    private fun cropByteArray(array: ByteArray, imageWidth: Int, cropRect: Rect): ByteArray {
        val croppedArray = ByteArray(cropRect.width() * cropRect.height())
        var i = 0
        array.forEachIndexed { index, byte ->
            val x = index % imageWidth
            val y = index / imageWidth
   
            if (cropRect.left <= x && x < cropRect.right && cropRect.top <= y && y < cropRect.bottom) {
                croppedArray[i] = byte
                i++
            }
        }
   
        return croppedArray
    }
   

First crop fun I took from here: Android: How to crop images using CameraX?

And I found also another crop fun, it seems that it is more complicated:

private fun cropByteArray(src: ByteArray, width: Int, height: Int, cropRect: Rect, ): ByteArray {
    val x = cropRect.left * 2 / 2
    val y = cropRect.top * 2 / 2
    val w = cropRect.width() * 2 / 2
    val h = cropRect.height() * 2 / 2
    val yUnit = w * h
    val uv = yUnit / 2
    val nData = ByteArray(yUnit + uv)
    val uvIndexDst = w * h - y / 2 * w
    val uvIndexSrc = width * height + x
    var srcPos0 = y * width
    var destPos0 = 0
    var uvSrcPos0 = uvIndexSrc
    var uvDestPos0 = uvIndexDst
    for (i in y until y + h) {
        System.arraycopy(src, srcPos0 + x, nData, destPos0, w) //y memory block copy
        srcPos0 += width
        destPos0 += w
        if (i and 1 == 0) {
            System.arraycopy(src, uvSrcPos0, nData, uvDestPos0, w) //uv memory block copy
            uvSrcPos0 += width
            uvDestPos0 += w
        }
    }
    return nData
}

Second crop fun I took from here: https://www.programmersought.com/article/75461140907/

I would be glad if someone can help improve the code.