Android 10 (api 29) camera2 api regression with wide-angle camera

Question

I'm using camera2 api in my camera app designed specifically for Google Pixel 3 XL. This device has two front facing cameras (wide-angle and normal). Thanks to multi-camera feature, I can access both physical camera devices simultaneously, and my app has a feature to toggle between those two cameras. Up until my recent upgrade to Android 10, I could accurately see two distinct results, but now my wide-angle capture frame has pretty much the same FOV (Field of View) as the normal camera one. So, the same code, same apk on Android 9 wide-angle capture result is wide, as expected, and after Andoird 10 upgrade - wide and normal cameras show practically identical FOV.

Here is a code snippet to demonstrate how I initialize both cameras and capture preview:

MainActivity.kt

 private val surfaceReadyCallback = object: SurfaceHolder.Callback {
        override fun surfaceChanged(p0: SurfaceHolder?, p1: Int, p2: Int, p3: Int) { }
        override fun surfaceDestroyed(p0: SurfaceHolder?) { }

        override fun surfaceCreated(p0: SurfaceHolder?) {

            // Get the two output targets from the activity / fragment
            val surface1 = surfaceView1.holder.surface  
            val surface2 = surfaceView2.holder.surface 

            val dualCamera = findShortLongCameraPair(cameraManager)!!
            val outputTargets = DualCameraOutputs(
                null, mutableListOf(surface1), mutableListOf(surface2))

            //Open the logical camera, configure the outputs and create a session
            createDualCameraSession(cameraManager, dualCamera, targets = outputTargets) { session ->

                val requestTemplate = CameraDevice.TEMPLATE_PREVIEW
                val captureRequest = session.device.createCaptureRequest(requestTemplate).apply {
                    arrayOf(surface1, surface2).forEach { addTarget(it) }
                }.build()

                session.setRepeatingRequest(captureRequest, null, null)
            }
        }
    }


    fun openDualCamera(cameraManager: CameraManager,
                       dualCamera: DualCamera,
                       executor: Executor = SERIAL_EXECUTOR,
                       callback: (CameraDevice) -> Unit) {

        cameraManager.openCamera(
            dualCamera.logicalId, executor, object : CameraDevice.StateCallback() {
                override fun onOpened(device: CameraDevice) { callback(device) }

                override fun onError(device: CameraDevice, error: Int) = onDisconnected(device)
                override fun onDisconnected(device: CameraDevice) = device.close()
            })
    }

    fun createDualCameraSession(cameraManager: CameraManager,
                                dualCamera: DualCamera,
                                targets: DualCameraOutputs,
                                executor: Executor = SERIAL_EXECUTOR,
                                callback: (CameraCaptureSession) -> Unit) {

        // Create 3 sets of output configurations: one for the logical camera, and
        // one for each of the physical cameras.
        val outputConfigsLogical = targets.first?.map { OutputConfiguration(it) }
        val outputConfigsPhysical1 = targets.second?.map {
            OutputConfiguration(it).apply { setPhysicalCameraId(dualCamera.physicalId1) } }
        val outputConfigsPhysical2 = targets.third?.map {
            OutputConfiguration(it).apply { setPhysicalCameraId(dualCamera.physicalId2) } }

        val outputConfigsAll = arrayOf(
            outputConfigsLogical, outputConfigsPhysical1, outputConfigsPhysical2)
            .filterNotNull().flatten()

        val sessionConfiguration = SessionConfiguration(SessionConfiguration.SESSION_REGULAR,
            outputConfigsAll, executor, object : CameraCaptureSession.StateCallback() {
                override fun onConfigured(session: CameraCaptureSession) = callback(session)
                override fun onConfigureFailed(session: CameraCaptureSession) = session.device.close()
            })


        openDualCamera(cameraManager, dualCamera, executor = executor) {
           it.createCaptureSession(sessionConfiguration)
        }
    }

DualCamera.kt Helper Class

data class DualCamera(val logicalId: String, val physicalId1: String, val physicalId2: String)

fun findDualCameras(manager: CameraManager, facing: Int? = null): Array<DualCamera> {
    val dualCameras = ArrayList<DualCamera>()

    manager.cameraIdList.map {
        Pair(manager.getCameraCharacteristics(it), it)
    }.filter {
        facing == null || it.first.get(CameraCharacteristics.LENS_FACING) == facing
    }.filter {
        it.first.get(CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES)!!.contains(
            CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERA)
    }.forEach {
        val physicalCameras = it.first.physicalCameraIds.toTypedArray()
        for (idx1 in 0 until physicalCameras.size) {
            for (idx2 in (idx1 + 1) until physicalCameras.size) {
                dualCameras.add(DualCamera(
                    it.second, physicalCameras[idx1], physicalCameras[idx2]))
            }
        }
    }

    return dualCameras.toTypedArray()
}

fun findShortLongCameraPair(manager: CameraManager, facing: Int? = null): DualCamera? {

    return findDualCameras(manager, facing).map {
        val characteristics1 = manager.getCameraCharacteristics(it.physicalId1)
        val characteristics2 = manager.getCameraCharacteristics(it.physicalId2)

        val focalLengths1 = characteristics1.get(
            CameraCharacteristics.LENS_INFO_AVAILABLE_FOCAL_LENGTHS) ?: floatArrayOf(0F)
        val focalLengths2 = characteristics2.get(
            CameraCharacteristics.LENS_INFO_AVAILABLE_FOCAL_LENGTHS) ?: floatArrayOf(0F)

        val focalLengthsDiff1 = focalLengths2.max()!! - focalLengths1.min()!!
        val focalLengthsDiff2 = focalLengths1.max()!! - focalLengths2.min()!!

        if (focalLengthsDiff1 < focalLengthsDiff2) {
            Pair(DualCamera(it.logicalId, it.physicalId1, it.physicalId2), focalLengthsDiff1)
        } else {
            Pair(DualCamera(it.logicalId, it.physicalId2, it.physicalId1), focalLengthsDiff2)
        }

        // Return only the pair with the largest difference, or null if no pairs are found
    }.sortedBy { it.second }.reversed().lastOrNull()?.first
}

And you can see the result on the attached screenshot, the top left corner one has much wider FOV than the same camera but running on Android 10

Is this a known regression with Android 10? Has anyone noticed similar behavior?

Answer 1

My understanding: I came across the same problem on my Pixel 3. It seems that the wide angle camera's frame has been cropped in the HAL layer before combination. Actually the FOV is not totally the same, as there is a little disparity between left and right camera. However, the default zoom level of wide camera seems to change according to the focal length.

But I could not find any official documentation about it. In Android 10, it claims improved the fusing of physical cameras: https://developer.android.com/about/versions/10/features#multi-camera

Solution:

If you wish to access the raw data from the wide angle front camera, you can create 2 camera sessions for both physical cameras instead of a single session for the logical camera.

Updated:

You can use the setPhysicalCameraKey to reset the zoom level https://developer.android.com/reference/android/hardware/camera2/CaptureRequest.Builder#setPhysicalCameraKey(android.hardware.camera2.CaptureRequest.Key%3CT%3E,%20T,%20java.lang.String)