Image is already closed. CameraX Analyzer

Question

I am trying to convert the ProxyImage from the cameraX analyzer to Bitmap to analyze the images using Tensor Flow light. So I implemented the cameraX Analyze call back which gives the image as proxyImage. That proxyImage i need to convert to bitmap. If I do this conversation on the UI thread, it makes the camera preview to lag. So I wanted to do it using Coroutines. Now the problem is whenever I pass the proxyImage to coroutines to convert it to bitmap on the background thread it crashes with the IllegalStateException that "Image is already closed."

08-04 16:28:59.690 16185-16185/com.example.camerax E/libEGL: call to OpenGL ES API with no current context (logged once per thread)
08-04 16:29:00.849 16185-16308/com.example.camerax E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-1
    Process: com.example.camerax, PID: 16185
    java.lang.IllegalStateException: Image is already closed
        at android.media.Image.throwISEIfImageIsInvalid(Image.java:68)
        at android.media.ImageReader$SurfaceImage$SurfacePlane.getBuffer(ImageReader.java:787)
        at androidx.camera.core.AndroidImageProxy$PlaneProxy.getBuffer(AndroidImageProxy.java:141)
        at com.example.camerax.MainActivity.getImageFromProxy(MainActivity.kt:216)
        at com.example.camerax.MainActivity.convertProxyImageToBitmap(MainActivity.kt:150)
        at com.example.camerax.MainActivity.access$convertProxyImageToBitmap(MainActivity.kt:38)
        at com.example.camerax.MainActivity$startCamera$3$1.invokeSuspend(MainActivity.kt:136)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)

I think next frame is de referencing the previous frame while app is converting the proxyImage to bitmap in the background thread. I read about the documentation and there they are saying that

Upon return from this method, the image reference is closed. Therefore, the method should complete analysis or make a copy instead of passing the image reference beyond the analysis method.

I am confused here that what it means by to make a copy of the image when we pass the image beyond the analysis method. How I can handle this scenario. Below is the code snippet.

  val imageAnalysisConfig = ImageAnalysisConfig.Builder()
            .setTargetResolution(Size(1280, 720))
            .build()
        val imageAnalysis = ImageAnalysis(imageAnalysisConfig)
        imageAnalysis.setAnalyzer { image: ImageProxy, _: Int ->

            classifier = Classifier.create(this, Classifier.Model.FLOAT, Classifier.Device.CPU, 1)

            CoroutineScope(Default).launch {
                convertProxyImageToBitmap(image)
            }
        }

Method to convert proxyImage to Bitmap.

private fun getImageFromProxy(image: ImageProxy): Bitmap {

        val yBuffer = image.planes[0].buffer // Y
        val uBuffer = image.planes[1].buffer // U
        val vBuffer = image.planes[2].buffer // V
        val ySize = yBuffer.remaining()
        val uSize = uBuffer.remaining()
        val vSize = vBuffer.remaining()
        val nv21 = ByteArray(ySize + uSize + vSize)
        //U and V are swapped
        yBuffer.get(nv21, 0, ySize)
        vBuffer.get(nv21, ySize, vSize)
        uBuffer.get(nv21, ySize + vSize, uSize)
        val yuvImage = YuvImage(nv21, ImageFormat.NV21, image.width, image.height, null)
        val out = ByteArrayOutputStream()
        yuvImage.compressToJpeg(Rect(0, 0, yuvImage.width, yuvImage.height), 100, out)
        val imageBytes = out.toByteArray()
        return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
    }

Thanks in advance for help.

Answer 1

I've recently encountered this problem. If I'm correct it means that you're doing something with image after image.close() has already been called. In your case it probably gets called automatically at the end of the lambda block within setAnalyzer { }, but when that happens you're still doing some asynchronous work with a coroutine.

You need to either remove the coroutine altogether or wrap it in a runBlocking { } block to wait for its completion, because otherwise the error won't go away (at least that's how I solved it recently). If you have set the back pressure strategy to STRATEGY_KEEP_ONLY_LATEST, you can theoretically spend all the time you want inside the block of code that performs your image analysis, since the frames currently produced by the camera will be thrown away instead of waiting needlessly.

If in the meantime you have updated your project to use the latest CameraX version and you're using a dedicated class, remember to always call image.close() yourself everytime you're returning out of your overridden analyze() method, because otherwise the camera preview on screen will freeze forever.

I am confused here that what it means by to make a copy of the image when we pass the image beyond the analysis method. How I can handle this scenario.

I think in this case you need to make a deep copy of image, meaning that you create a new instance and set all of its content and internal state to those of the original one instead of just doing a simple assignment of the reference.