How to increase frame rate with Android CameraX ImageAnalysis?

Question

I'm investigating the new CameraX API, in relation to how viable it might be to switch over from our current Camera2 system.

In our Camera2 system, we use an OpenGL surface to capture the frames from a PreviewCaptureSession, and we are hitting consistent 30fps image processing speeds across most devices, with some able to hit 60fps with AutoExposure settings enabled.

CameraX isn't giving anything near that speed and I'm not sure if its something I've missed in setup.

I've setup the test examples for CameraX and ImageAnalysis, but I'm getting locked frame rates for the number of images that comes through.

For example, I could set the resolution as low as 320x240 up to 1920x960 and both will come out at a (seemingly capped) 16fps.

When I add a Preview usecase to run along side it, and set enableTorch(true), the ImageAnalysis usecase will suddenly start getting more like 20fps, with it occasionally peaking up to 30ish.

Clearly the Preview usecase alters some of the AutoExposure state of the camera?

Here's a snipper of my current setup...

 private fun startCameraAnalysis() {
        val metrics = DisplayMetrics().also { viewFinder.display.getRealMetrics(it) }
        var resolution = Size(metrics.widthPixels, metrics.heightPixels)
        resolution = Size(640, 480) //set to fixed size for testing

        val aspectRatio = Rational(resolution.width, resolution.height)
        val rotation = viewFinder.display.rotation

        // Setup image analysis pipeline
        val analyzerConfig = ImageAnalysisConfig.Builder().apply {
            val analyzerThread = HandlerThread(
                "LuminosityAnalysis").apply { start() }
            setCallbackHandler(Handler(analyzerThread.looper))
     setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
            setTargetRotation(rotation)
            setTargetAspectRatio(aspectRatio)
            setTargetResolution(resolution)
        }.build()

        // Setup preview pipeline
        val previewConfig = PreviewConfig.Builder().apply {
            setTargetRotation(rotation)
            setTargetAspectRatio(aspectRatio)
            setTargetResolution(resolution)
        }.build()

        // Build Preview useCase
        val preview = Preview(previewConfig)
        preview.enableTorch(true)

        // Build Analysis useCase
        val analyzer = ImageAnalysis(analyzerConfig)
        analyzer.analyzer = LuminosityAnalyzer()

        CameraX.bindToLifecycle(this, preview, analyzer )
        preview.enableTorch(true)
}

Is there anyway to alter the camera settings in CameraX around the ImageAnalysis to get higher frame rates?

Is there anyway at all in fact to alter things like Sensor Duration, ISO, exposure?

Answer 1

So I've spent some more time investigating and I think I've come up with a solution for now.

It turns out, the ImageAnalysisConfig is not extendable, so you can't alter the camera configuration when just using one of those, so the default camera settings will be used which on my phone I think resulted in AE being on and hitting 16ish FPS.

If you spin up a PreviewConfig to run along side it at the same time, you can then extend this with a Camera2Config.Extender and alter the camera2 properties directly. This can increase the camera preview frame rate, and the Analyser will also start getting frames at the same rate.

So for example, I add this to my PreviewConfig...

    // Create Camera2 extender
    var camera2Extender = Camera2Config.Extender(previewConfig)
        .setCaptureRequestOption(CaptureRequest.CONTROL_MODE, CaptureRequest.CONTROL_MODE_OFF)
        .setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF)
        .setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_OFF)
        .setCaptureRequestOption(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_OFF)
        .setCaptureRequestOption(CaptureRequest.FLASH_MODE, CaptureRequest.FLASH_MODE_TORCH)
        .setCaptureRequestOption(CaptureRequest.SENSOR_SENSITIVITY, 100)
        .setCaptureRequestOption(CaptureRequest.SENSOR_FRAME_DURATION, 16666666)
        .setCaptureRequestOption(CaptureRequest.SENSOR_EXPOSURE_TIME, 20400000)

So this started hitting 30fps fine in the ImageAnalyser.

If I want to hit 60, I can set...

.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(60,60))

Obviously assuming the device support (60,60) target FPS range.

So it seems the full Camera2 logic is still available in CameraX, its just a bit clunky that its a little hidden away in a Camera2Config extender, and this only works with Preview use cases.

Answer 2

Alright, this was driving me crazy for hours.

Expanding upon Ian's answer for the newest version of CameraX, you can now extend ImageAnalysis directly. See CameraX equivalent of Camera2's CaptureRequest

So to get 60FPS, we can use this modified code (example in Java and Kotlin):

// Java

ImageAnalysis.Builder builder = new ImageAnalysis.Builder();
Camera2Interop.Extender ext = new Camera2Interop.Extender<>(builder);
ext.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF);
ext.setCaptureRequestOption(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, new Range<Integer>(60, 60));
ImageAnalysis imageAnalysis = builder.build();

// Kotlin

val builder = ImageAnalysis.Builder()
val ext: Camera2Interop.Extender<*> = Camera2Interop.Extender(builder)
ext.setCaptureRequestOption(
                CaptureRequest.CONTROL_AE_MODE,
                CaptureRequest.CONTROL_AE_MODE_OFF
            )
ext.setCaptureRequestOption(
                CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE,
                Range<Int>(60, 60)
            )
val imageAnalysis = builder.build()