Android: Camera Asynctask with Preview Callback

Question

I've managed to get camera preview with custom filter (grayscale, hue, etc) working. This custom filter is applied with preview callback by manipulating the array of RGB and then drawing it back to a canvas then display it at the surface view.

The drawback from this is I get a very low FPS. With this low FPS, it's doing too much work in the UI thread if I don't do this in the background thread using Asynctask. So I tried to use Asynctask for the camera operation (my main purpose is to get the UI still working perfectly even with the heavy work from the camera preview callback).

But even after I used Asynctask, it didn't help much. So I am wondering is it my implementation is wrong or is it because even with asynctask the UI thread will still be affected?

Snippet of my code is below:

CameraActivity.java

public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    Log.d("ACTIVITY_LIFECYCLE","CameraActivity: onCreate");
    setContentView(R.layout.camera_layout);
}

@TargetApi(Build.VERSION_CODES.HONEYCOMB)
@Override
protected void onResume() {
    Log.d("ACTIVITY_LIFECYCLE","CameraActivity: onResume");
    if(preview == null){
        preview = new CameraPreviewAsync(this,camera);
        preview.execute();
    }
    super.onResume();
}

@Override
protected void onPause() {
    Log.d("ACTIVITY_LIFECYCLE","CameraActivity: onPause");
    if(preview!=null){
        preview.cancel(true);
        camera = preview.getCamera();
        if(camera!=null){
            camera.stopPreview();
            camera.setPreviewCallback(null);
            camera.release();
            camera = null;
            preview.setCamera(camera);
        }
        preview = null;
    }
    super.onPause();
}

@Override
public void onDestroy(){
    Log.d("ACTIVITY_LIFECYCLE","CameraActivity: onDestroy");
    super.onDestroy();
} 

CameraPreviewAsync.java:

private final String TAG = "CameraPreviewAsync";

private CameraActivity camAct;
private Camera mCamera;
private int cameraId;
private SurfaceView mSurfaceView;
private SurfaceHolder mHolder;

private boolean isPreviewRunning = false;
private int[] rgbints;
private int width;
private int height;
private Bitmap mBitmap;

public CameraPreviewAsync(CameraActivity act, Camera cam){
    this.camAct = act;
    this.mCamera = cam;
    this.mSurfaceView = (SurfaceView) act.findViewById(R.id.surfaceView);
}

public void resetSurface(){
    if(mCamera!=null){
        mCamera.stopPreview();
        mCamera.setPreviewCallback(null);
        mCamera.release();
        mCamera = null;
    }
    int tempId = R.id.surfaceView;
    RelativeLayout buttonBar = (RelativeLayout) camAct.findViewById(R.id.buttonBar);
    ((RelativeLayout) camAct.findViewById(R.id.preview)).removeAllViews();

    SurfaceView newSurface = new SurfaceView(camAct);
    newSurface.setId(tempId);
    RelativeLayout.LayoutParams layParams = new RelativeLayout.LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT);
    layParams.alignWithParent = true;
    newSurface.setLayoutParams(layParams);
    ((RelativeLayout) camAct.findViewById(R.id.preview)).addView(newSurface);
    ((RelativeLayout) camAct.findViewById(R.id.preview)).addView(buttonBar);
}

@Override
protected void onPreExecute() {
    //Things to do before doInBackground executed
    Log.d(TAG,"onPreExecute");

    RelativeLayout.LayoutParams layParams = new RelativeLayout.LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT);
    layParams.alignWithParent = true;
    mSurfaceView.setLayoutParams(layParams);

    //Check number of camera in the device, if less than 2 then remove swap button
    if (Camera.getNumberOfCameras() < 2) {
        ((RelativeLayout) camAct.findViewById(R.id.buttonBar)).removeViewAt(R.id.cameraSwap);
    }

    //Opening the camera
    cameraId = findBackFacingCamera();
    if (cameraId < 0) {
        cameraId = findFrontFacingCamera();
        if (cameraId < 0)
            Toast.makeText(camAct, "No camera found.", Toast.LENGTH_LONG).show();
        else
            mCamera = Camera.open(cameraId);
    } else {
        mCamera = Camera.open(cameraId);
    }

    //invalidate the menu bar and show menu appropriately
    camAct.invalidateOptionsMenu();

    // get Camera parameters and set it to Auto Focus
    if(mCamera!=null){
        Camera.Parameters params = mCamera.getParameters();
        List<String> focusModes = params.getSupportedFocusModes();
        if (focusModes.contains(Camera.Parameters.FOCUS_MODE_AUTO)) {
            // set the focus mode
            params.setFocusMode(Camera.Parameters.FOCUS_MODE_AUTO);
            // set Camera parameters
            mCamera.setParameters(params);
        }
    }

    super.onPreExecute();
}

@Override
protected Void doInBackground(Void... params) {
    //Things to do in the background thread
    Log.d(TAG,"doInBackground");

    mHolder = mSurfaceView.getHolder();
    mHolder.addCallback(surfaceCallback);

    return null;
}      

@Override
protected void onPostExecute(Void values) {
    //Things to do after doInBackground
    Log.d(TAG,"onPostExecute");

}

@Override
protected void onCancelled(){
    super.onCancelled();
}

/*
 * ************************************************************************************
 * SURFACEHOLDER CALLBACK
 * ************************************************************************************
 */
SurfaceHolder.Callback surfaceCallback = new SurfaceHolder.Callback() {

    @Override
    public void surfaceCreated(SurfaceHolder holder) {
        Log.d(TAG,"surfaceCreated!!");
        if(CameraActivity.filterMode == CameraActivity.NORMAL_FILTER){
            try {
                if (mCamera != null) {
                    mCamera.startPreview();
                    mCamera.setPreviewDisplay(holder);
                }else{
                    Log.d(TAG,"CAMERA IS NULL in surfaceCreated!!");
                }
            } catch (IOException exception) {
                Log.e(TAG, "IOException caused by setPreviewDisplay()", exception);
            }   
        }else{
            synchronized(mSurfaceView){
                if(isPreviewRunning){
                    return;
                }else{                      

                    mSurfaceView.setWillNotDraw(false);
                    if(mCamera!=null){
                        isPreviewRunning = true;
                        Camera.Parameters p = mCamera.getParameters();
                        List<Size> sizes = p.getSupportedPreviewSizes();

                        Size size = p.getPreviewSize();
                        width = size.width;
                        height = size.height;

                        p.setPreviewFormat(ImageFormat.NV21);
                        showSupportedCameraFormats(p);
                        mCamera.setParameters(p);

                        rgbints = new int[width * height];

                        mCamera.startPreview();
                        mCamera.setPreviewCallback(previewCallback);
                    }
                }
            }
        }
    }

    @Override
    public void surfaceDestroyed(SurfaceHolder holder) {
        Log.d(TAG,"surfaceDestroyed!");

        if(CameraActivity.filterMode == CameraActivity.NORMAL_FILTER){
            if (mCamera != null) {
                mCamera.stopPreview();
                isPreviewRunning = false;
            }
        }else{
            synchronized(mSurfaceView){
                if(mCamera!=null){
                    mCamera.setPreviewCallback(null);
                    mCamera.stopPreview();
                    isPreviewRunning = false;
                }
            }
        }
    }

    @Override
    public void surfaceChanged(SurfaceHolder holder, int format, int width,
            int height) {
        Log.d(TAG,"surfaceChanged!");
    }
};


/*
 * ************************************************************************************
 * CAMERA PREVIEW CALLBACK
 * ************************************************************************************
 */

Camera.PreviewCallback previewCallback = new Camera.PreviewCallback() {

    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        if (!isPreviewRunning)
            return;
        Canvas resCanvas = null;

        if (mHolder == null) {
            return;
        }

        try {
            synchronized (mHolder) {
                resCanvas = mHolder.lockCanvas(null);
                int resCanvasW = resCanvas.getWidth();
                int resCanvasH = resCanvas.getHeight();

                if(mBitmap == null){
                    mBitmap =  Bitmap.createBitmap (width, height, Bitmap.Config.ARGB_8888);
                }

                decodeYUV(rgbints, data, width, height);

                Canvas canvas = new Canvas(mBitmap);

                //Setting the filter
                if(camAct.getCustomFilter().equalsIgnoreCase("NORMAL")) ;//don't change the rgb value
                if(camAct.getCustomFilter().equalsIgnoreCase("GRAYSCALE")) rgbints = grayscale(rgbints);
                if(camAct.getCustomFilter().equalsIgnoreCase("INVERT")) rgbints = invert(rgbints);
                if(camAct.getCustomFilter().equalsIgnoreCase("BOOSTRED")) rgbints = boostColor(rgbints,1);
                if(camAct.getCustomFilter().equalsIgnoreCase("BOOSTGREEN")) rgbints = boostColor(rgbints,2);
                if(camAct.getCustomFilter().equalsIgnoreCase("BOOSTBLUE")) rgbints = boostColor(rgbints,3);
                if(camAct.getCustomFilter().equalsIgnoreCase("NOISE")) rgbints = noise(rgbints);
                if(camAct.getCustomFilter().equalsIgnoreCase("HUE")) rgbints = hue(rgbints);
                if(camAct.getCustomFilter().equalsIgnoreCase("SATURATION")) rgbints = saturation(rgbints);
                if(camAct.getCustomFilter().equalsIgnoreCase("ENGRAVE")) rgbints = engrave(rgbints);
                if(camAct.getCustomFilter().equalsIgnoreCase("EMBOSS")) rgbints = emboss(rgbints);

                // draw the decoded image, centered on canvas
                canvas.drawBitmap(rgbints, 0, width, 0,0, width, height, false, null);

                resCanvas.drawBitmap (mBitmap, resCanvasW-((width+resCanvasW)>>1), resCanvasH-((height+resCanvasH)>>1),null);
            }
        }  catch (Exception e){
            e.printStackTrace();
        } finally {
            // do this in a finally so that if an exception is thrown
            // during the above, we don't leave the Surface in an
            // inconsistent state
            if (resCanvas != null) {
                mHolder.unlockCanvasAndPost(resCanvas);
            }
        }
    }
};

Any help is much appreciated! :) Thanks in advance guys!

Answer 1

Perhaps my answer is too late for you, but I was researching the same topic so thought I'd share my findings anyway...

First of all, if Camera's "open" is called on the AsyncTask and then that thread exists and seizes to exist - we can't really expect callbacks to be coming from it, can we. So, if we want callbacks - then we need to have a thread that lives for at least as long as we want our callbacks.

But wait, there is more... Documentation for Camera.PreviewCallback is not the clearest, but one of the bad hints is this "This callback is invoked on the event thread open(int) was called from." What do they mean by "event" thread? Well, it's not very clear - but looking at the Android code and experimenting - what they need is a thread that contains a Looper. Probably too much details but in the Camera constructor (which is called from open method) there is code that tries to first obtain a Looper of the current thread, if that doesn't exist - it tries to get main thread looper - which lives on UI thread. Camera then uses a Handler to dispatch callbacks, and other methods via the looper that it initializes like that. Now you can probably see why you were getting your call backs on the main thread even though you opened camera from a different thread - your worker thread didn't have a looper - so Camera defaulted to using the main one.

I got callbacks working from my worker thread for which I used HandlerThread in a method along these lines:

private void startCamera() {
    if (mCameraThread == null) {
        mCameraThread = new HandlerThread(CAMERA_THREAD_NAME);
        mCameraThread.start();
        mCameraHandler = new Handler(mCameraThread.getLooper());
    }
    mCameraHandler.post(new Runnable() {
        @Override
        public void run() {
            try {
                mCamera = Camera.open();

...

I used debugger to confirm that my onPreviewFrame did run on the worker thread. I also had animation running on UI thread, which was jerky before I transitioned frame processing away from the main thread but now is as smooth as butter.

Do note, that if you kill your worker thread, then of course your callbacks will stop too and Camera (well rather Handler) will complain that you try to use dead thread.

BTW, as an alternative solution one of course could have had callbacks be called on the main thread, but processing of frame data could be delegated to a separate thread.