Android draw on camera preview

Question

I'm making a virtual reality application where the camera should detect faces, locate them and show their location on the camera preview.

I know of 3 ways to do it, I'd like to use GLSurfaceView to be as fast as possible (according to this post), but currently I'm trying to draw on the same SurfaceView where the camera is using for its preview. My callback to draw on it would be onFaceDetection like so:

public class MyActivity extends Activity implements SurfaceHolder.Callback, Camera.FaceDetectionListener {
    Camera camera;
    SurfaceView svPreview;
    SurfaceHolder previewHolder;
    TextView tvInfo;
    Paint red;

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        svPreview = (SurfaceView) findViewById(R.id.svPreview);
        tvInfo = (TextView) findViewById(R.id.tvInfo);

        red = new Paint();
        red.setStyle(Paint.Style.STROKE);
        red.setStrokeWidth(3);

        previewHolder = svPreview.getHolder();
        previewHolder.addCallback(this);
        previewHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
    }

    public void surfaceCreated(SurfaceHolder arg0) {
        camera = Camera.open();
        try {
            camera.setDisplayOrientation(90);
            camera.setFaceDetectionListener(this);
            camera.setPreviewDisplay(previewHolder);
        }
        catch (IOException e) {
            e.printStackTrace();
        }
    }

    public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
        // . . .
        camera.startPreview();
        camera.autoFocus(null);
        camera.startFaceDetection();
    }

    public void surfaceDestroyed(SurfaceHolder arg0) {
        camera.stopFaceDetection();
        camera.cancelAutoFocus();
        camera.stopPreview();
        camera.release();
        camera = null;
    }

    public void onFaceDetection(Face[] faces, Camera camera) {  
        tvInfo.setText("Faces: " + String.valueOf(faces.length));

        Canvas canvas = previewHolder.lockCanvas();
        for(int i=0; i < faces.length; i++) {
            Point leftEye = faces[i].leftEye;
            Point rightEye = faces[i].rightEye;
            // this is not working
            canvas.drawPoint(leftEye.x, leftEye.y, red);
        }
        previewHolder.unlockCanvasAndPost(canvas);
    }
}

With this code I keep getting this error:

09-03 19:35:42.743: E/SurfaceHolder(19394): Exception locking surface
09-03 19:35:42.743: E/SurfaceHolder(19394): java.lang.IllegalArgumentException
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.view.Surface.lockCanvasNative(Native Method)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.view.Surface.lockCanvas(Surface.java:76)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.view.SurfaceView$4.internalLockCanvas(SurfaceView.java:744)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.view.SurfaceView$4.lockCanvas(SurfaceView.java:720)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at com.bluetooth.activities.MyActivity.onFaceDetection(MyActivity.java:90)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.hardware.Camera$EventHandler.handleMessage(Camera.java:729)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.os.Handler.dispatchMessage(Handler.java:99)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.os.Looper.loop(Looper.java:137)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.app.ActivityThread.main(ActivityThread.java:4424)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at java.lang.reflect.Method.invokeNative(Native Method)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at java.lang.reflect.Method.invoke(Method.java:511)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:784)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:551)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at dalvik.system.NativeStart.main(Native Method)
09-03 19:35:42.743: W/dalvikvm(19394): threadid=1: thread exiting with uncaught exception (group=0x40a561f8)
09-03 19:35:42.766: E/AndroidRuntime(19394): FATAL EXCEPTION: main
09-03 19:35:42.766: E/AndroidRuntime(19394): java.lang.IllegalArgumentException
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.view.Surface.unlockCanvasAndPost(Native Method)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.view.SurfaceView$4.unlockCanvasAndPost(SurfaceView.java:775)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at com.bluetooth.activities.MyActivity.onFaceDetection(MyActivity.java:99)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.hardware.Camera$EventHandler.handleMessage(Camera.java:729)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.os.Handler.dispatchMessage(Handler.java:99)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.os.Looper.loop(Looper.java:137)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.app.ActivityThread.main(ActivityThread.java:4424)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at java.lang.reflect.Method.invokeNative(Native Method)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at java.lang.reflect.Method.invoke(Method.java:511)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:784)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:551)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at dalvik.system.NativeStart.main(Native Method)

Is there a problem with the camera trying to draw its preview on the same SurfaceView the face detection callback? How can I do this without layering SurfaceViews?

Answer 1

You can't lock and draw on a SurfaceView which has Type.PUSH_BUFFERS, (the one you're displaying frames to). You have to create another view above your original one in the Z direction and draw on a SurfaceView in that View.

So in your main.xml create a custom view below your original view in a FrameLayout.

Create and handle a SurfaceView in your Activity View. Add this view to the Camera preview display. Start your custom view passing the SurfaceHolder. In this view you can lock and draw on a canvas.

Answer 2

As James pointed out you need to create custom surface which extends SurfaceView (I usually implement SurfaceHolder.Callback also):

public class CameraSurfacePreview extends SurfaceView implements SurfaceHolder.Callback

Constructor will be something like:

public CameraSurfacePreview(Context context) {
     super(context);
     ...
     mHolder = getHolder();
     mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
     ...

You need to bind camera with your surface after camera open call (if you implement implements SurfaceHolder.Callback put this somewhere inside overridden surfaceCreated):

mCamera = Camera.open();
mCamera.setPreviewDisplay(mHolder);

Finally you need to add instance of your custom surface somehere in activity content view:

CameraSurfacePreview cameraSurfacePreview = new CameraSurfacePreview(this);
//camera surface preview is first child!
((ViewGroup)findViewById(R.id.cameraLayout)).addView(cameraSurfacePreview, 0); 

In my example layout for activity looks something like(I am showing camera preview in main frame layout):

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android" 
              android:layout_width="fill_parent"        
                      android:layout_height="fill_parent" 
              android:layout_gravity="top|left"
              android:id="@+id/cameraLayout">