Camera2: Make SurfaceView aspect fit to fill (scaleType CENTER_CROP)

Question

I've been following the Camera2 example (android/camera-samples/Camera2Video) to create an abstraction over the Camera2 library. The idea of my abstraction is to give the user the ability to use a Camera view from React Native, so it's just a <Camera> view which can be whatever size/aspect ratio.

While this works out of the box on iOS, I can't seem to get the preview on Android to display "what the camera sees" in the correct aspect ratio.

The official example from Android works like this:

• They create a custom SurfaceView extension that should automatically fit to the correct aspect ratio (see: AutoFitSurfaceView)
• They use that AutoFitSurfaceView in their layout
• They add a listener to the AutoFitSurfaceView's Surface to find out when it has been created (source)
• Once the Surface has been created, they call getPreviewOutputSize(...) to get the best matching camera preview size (e.g. so you don't stream 4k for a 1080p screen, that's wasted pixels)
• Then they pass the best matching camera preview size to the AutoFitSurfaceView::setAspectRatio(...) function
• By knowing the desired aspect ratio, the AutoFitSurfaceView should then automatically perform a center-crop transform in it's onMeasure override

If you read the source code of their getPreviewOutputSize(...) function, you might notice that this uses a Display to find the best matching preview size. If I understood the code correctly, this would only work if the camera preview (AutoFitSurfaceView) is exactly the same size as the device's screen. This is poorly designed, as there are lots of cases where that simply isn't true. In my case, the Camera has a bit of a bottom spacing/margin, so it doesn't fill the screen and therefore has weird resolutions (1080x1585 on a 1080x1920 screen)

With that long introduction, here comes my question: How do I actually perform a correct center crop transform (aka scaleType = CENTER_CROP) on my SurfaceView? I've tried the following:

• Set the size of my SurfaceView using SurfaceHolder::setFixedSize(...), but that didn't change anything at all
• Remove their getPreviewOutputSize(...) stuff and simply use the highest resolution available
• Use the Android View properties scaleX and scaleY to scale the view by the aspect-ratio difference of the view <-> camera input scaler size (this somewhat worked, but is giving me errors if I try to use high-speed capture: Surface size 1080x1585 is not part of the high speed supported size list [1280x720, 1920x1080])

Any help appreciated!

Answer 1

A SurfaceView is a View that also holds a Surface. The Surface is the buffer that the camera needs to output the captured images. The 'View` is what makes it shown in the screen. See the section about SurfaceTexture in Android's Graphic architecture (actually, the whole collection of articles is interesting).

Introduction To control the aspect ratio of the displayed camera preview, you have to consider two factors:

• The default behavior of the View is to stretch the Surface to it occupies the whole available space. View offers several properties to change this behavior. The most convenient to control aspect ratio are scaleX and scaleY. Both work the same way - scaleX = 1 means that the surface is stretched to cover the whole width of the view; scaleX = 0.5f means that the surface is stretched to cover 1/2 of the width. Same applies for scaleY.
• When creating a camera session, and later when requesting captures to the session, you have to specify the list of target surfaces. The capture procedure checks the sizes of the target surfaces, and chooses the best match among camera's available resolutions. If you want to be sure of the selected resolution, you need to set the surface size to be an exact match.

Out of scope In the examples below:

• SelectedCameraDescription contains some characteristics of the selected camera. Chiefly among them, the selected resolution. Android's article Camera lenses and capabilities shows some examples on this topic.
• activateCamera(surface: Surface) is a method that initializes a camera session and starts a capture request. Android's article Camera capture sessions and requests gives some hints on this part.
• Also, I kept the whole business of requesting permissions outside of the example.

Display the camera preview with correct aspect ratio

1. Display the SurfaceView object, and call a method to start the capture session as soon as the surface is available. As it is not a @Composable object, you need to embed it into an AndroidView so as to obtain the Context, and use it to create a new instance of SurfaceView. The code below will already display the preview, although not with the correct ratio.

@Composable
fun ShowCamera(
    cameraDescription: SelectedCameraDescription,
    activateCamera: (Surface) -> Unit,
    modifier: Modifier = Modifier) {

    AndroidView(
        modifier = modifier,
        factory = { context ->
            SurfaceView(context).apply {
                this.post {
                    this.setBackgroundColor(Color.TRANSPARENT)
                    activateCamera(this.holder.surface)
                }
            }
        }
    )
}

2. Let's extend the surface view into a CameraPreviewSurfaceView so we can have a convenient place to include the code to scale the camera images. The custom view has two tasks. First is to set the size of the surface to the exact size of the resolution we want the camera session to select. This is done in the init method. Second is to scale the surface so it shows on the view with the correct aspect ratio. This is done in the onMeasure method. This method is invoked by the View whenever it needs to adapt itself to the surrounding components. Typically when first inflating the view, or when rotating the device.

/**
 * A custom [SurfaceView], adapted for camera preview.
 */
@SuppressLint("ViewConstructor") // No need of view constructor, as we're using jetpack.
class CameraPreviewSurfaceView(
    private val cameraDescription: SelectedCameraDescription,
    context: Context): SurfaceView(context) {

    /**
     * Sets the size of the inner surface.
     * The camera session picks the camera resolution from the size of the surface, so it
     * is important to match one of the available resolutions.
     */
    init {
        this.holder.setFixedSize(
            cameraDescription.sizeInPixels.width,
            cameraDescription.sizeInPixels.height)
    }

    /**
     * Recalculates the scale required for the camera output to be displayed on the
     * view without distortion.
     */
    override fun onMeasure(widthMeasureSpec: Int, heightMeasureSpec: Int) {
        // Don't interfere with the normal measure procedure:
        super.onMeasure(widthMeasureSpec, heightMeasureSpec)

        // Difference between camera orientation and device orientation:
        val cameraOrientation = cameraDescription.orientation
        val deviceOrientation = deviceOrientation()
        val rotation = cameraOrientation - deviceOrientation

        // Depending on the rotation, the camera width and height are inverted:
        val cameraOutputWidth: Int
        val cameraOutputHeight: Int
        if (rotation % 180 == 0) {
            cameraOutputWidth = cameraDescription.sizeInPixels.width
            cameraOutputHeight = cameraDescription.sizeInPixels.height
        } else {
            cameraOutputWidth = cameraDescription.sizeInPixels.height
            cameraOutputHeight = cameraDescription.sizeInPixels.width
        }

        // Camara output aspect and measured view aspect:
        val cameraOutputAspect =
            cameraOutputWidth.toFloat() / cameraOutputHeight.toFloat()
        val measuredAspect =
            measuredWidth.toFloat() / measuredHeight.toFloat()

        // Surface will be stretched to completely fit the measured area.
        // We want the scale to do the inverse:
        val scaleAspect = cameraOutputAspect / measuredAspect

        if (scaleAspect < 1) {
            scaleX = scaleAspect
            scaleY = 1f
        } else {
            scaleX = 1f
            scaleY = 1f / scaleAspect
        }
    }

    /**
     * Obtains the current orientation of the context,
     * and maps it degrees.
     */
    private fun deviceOrientation(): Int {
        val displayRotation = ContextCompat.getDisplayOrDefault(context).rotation
        return when(displayRotation) {
            Surface.ROTATION_0 -> 0
            Surface.ROTATION_90 -> 90
            Surface.ROTATION_180 -> 180
            Surface.ROTATION_270 -> 270
            else -> 0
        }
    }
}

3. Call the custom surface view from the user interface:

@Composable
fun ShowCamera(
    cameraDescription: SelectedCameraDescription,
    activateCamera: (Surface) -> Unit,
    modifier: Modifier = Modifier) {

    AndroidView(
        modifier = modifier,
        factory = { context ->
            CameraPreviewSurfaceView(cameraDescription, context).apply {
                this.post {
                    this.setBackgroundColor(Color.TRANSPARENT)
                    activateCamera(this.holder.surface)
                }
            }
        }
    )
}

Some additional points

• There are alternatives to SurfaceView: for example TextureView and CameraViewfinder. See Android's Camera preview. Also find there a different scaling calculation. If you do the numbers you'll realize that the results are the same, but maybe you prefer Android's code than mine's.
• Anything drawn on the SurfaceView is displayed over the actual Surface. For example, it is important that the background is set to transparent. As TRANSPARENT is the default background color, you don't actually need to set it, unless you set another color previously.