Android Camera2 API Showing Processed Preview Image

Question

New Camera 2 API is very different from old one.Showing the manipulated camera frames to user part of pipeline is confuses me. I know there is very good explanation on Camera preview image data processing with Android L and Camera2 API but showing frames is still not clear. My question is what is the way of showing frames on screen which came from ImageReaders callback function after some processing while preserving efficiency and speed in Camera2 api pipeline?

Example Flow :

camera.add_target(imagereader.getsurface) -> on imagereaders callback do some processing -> (show that processed image on screen?)

Workaround Idea : Sending bitmaps to imageview every time new frame processed.

Answer 1

I had the same need, and wanted a quick and dirty manipulation for a demo. I was not worried about efficient processing for a final product. This was easily achieved using the following java solution.

My original code to connect the camera2 preview to a TextureView was commented-out and replaced with a surface to an ImageReader:

    // Get the surface of the TextureView on the layout
    //SurfaceTexture texture = mTextureView.getSurfaceTexture();
    //if (null == texture) {
    //    return;
    //}
    //texture.setDefaultBufferSize(mPreviewWidth, mPreviewHeight);
    //Surface surface = new Surface(texture);

    // Capture the preview to the memory reader instead of a UI element
    mPreviewReader = ImageReader.newInstance(mPreviewWidth, mPreviewHeight, ImageFormat.JPEG, 1);
    Surface surface = mPreviewReader.getSurface();

    // This part stays the same regardless of where we render
    mCaptureRequestBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
    mCaptureRequestBuilder.addTarget(surface);
    mCameraDevice.createCaptureSession(...

Then I registered a listener for the image:

mPreviewReader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
    @Override
    public void onImageAvailable(ImageReader reader) {
        Image image = reader.acquireLatestImage();
        if (image != null) {
            Image.Plane plane = image.getPlanes()[0];
            ByteBuffer buffer = plane.getBuffer();
            byte[] bytes = new byte[buffer.capacity()];
            buffer.get(bytes);
            Bitmap preview = BitmapFactory.decodeByteArray(bytes, 0, buffer.capacity());
            image.close();
            if(preview != null ) {
                // This gets the canvas for the same mTextureView we would have connected to the
                // Camera2 preview directly above.
                Canvas canvas = mTextureView.lockCanvas();
                if (canvas != null) {
                    float[] colorTransform = {
                            0, 0, 0, 0, 0,
                            .35f, .45f, .25f, 0, 0,
                            0, 0, 0, 0, 0,
                            0, 0, 0, 1, 0};
                    ColorMatrix colorMatrix = new ColorMatrix();
                    colorMatrix.set(colorTransform); //Apply the monochrome green
                    ColorMatrixColorFilter colorFilter = new ColorMatrixColorFilter(colorMatrix);
                    Paint paint = new Paint();
                    paint.setColorFilter(colorFilter);
                    canvas.drawBitmap(preview, 0, 0, paint);
                    mTextureView.unlockCanvasAndPost(canvas);
                }
            }
        }
    }
}, mBackgroundPreviewHandler);

Answer 2

Edit after clarification of the question; original answer at bottom

Depends on where you're doing your processing.

If you're using RenderScript, you can connect a Surface from a SurfaceView or a TextureView to an Allocation (with setSurface), and then write your processed output to that Allocation and send it out with Allocation.ioSend(). The HDR Viewfinder demo uses this approach.

If you're doing EGL shader-based processing, you can connect a Surface to an EGLSurface with eglCreateWindowSurface, with the Surface as the native_window argument. Then you can render your final output to that EGLSurface and when you call eglSwapBuffers, the buffer will be sent to the screen.

If you're doing native processing, you can use the NDK ANativeWindow methods to write to a Surface you pass from Java and convert to an ANativeWindow.

If you're doing Java-level processing, that's really slow and you probably don't want to. But can use the new Android M ImageWriter class, or upload a texture to EGL every frame.

Or as you say, draw to an ImageView every frame, but that'll be slow.

---

Original answer:

If you are capturing JPEG images, you can simply copy the contents of the ByteBuffer from Image.getPlanes()[0].getBuffer() into a byte[], and then use BitmapFactory.decodeByteArray to convert it to a Bitmap.

If you are capturing YUV_420_888 images, then you need to write your own conversion code from the 3-plane YCbCr 4:2:0 format to something you can display, such as a int[] of RGB values to create a Bitmap from; unfortunately there's not yet a convenient API for this.

If you are capturing RAW_SENSOR images (Bayer-pattern unprocessed sensor data), then you need to do a whole lot of image processing or just save a DNG.