Fighting with SurfaceView, Camera and OpenGL

Question

We've been fighting with some problems relating to SurfaceViews for more than a week, and find no proper solution to them. We read the other questions in the forum regarding to similar problems (and even Mixare source code) but couldn't find an answer, so we hope you could help us somehow.

Scenario: We've got

• a SurfaceView for the Camera
• a SurfaceView for an OpenGL layer, which goes on top of the camera.
• another View, which shows some information about what we can see on the screen. This one goes on top of both SurfaceViews.

Problem:

No matter how hard we try, both SurfaceViews apparently don't get on well with each other. If we try to:

setContentView(mCameraPreview); 
addContentView(mGLSurfaceView, new LayoutParams(LayoutParams.WRAP_CONTENT, LayoutParams.WRAP_CONTENT)); 
addContentView(mInfoView, new LayoutParams(LayoutParams.WRAP_CONTENT, LayoutParams.WRAP_CONTENT));

(which seems logical), everything goes as expected until we lock/unlock the phone. After that, the GLSurfaceView just disappears (not the InfoView, that one is still shown).

If, instead, we try to:

setContentView(mGLSurfaceView); 
addContentView(mCameraPreview, new LayoutParams(LayoutParams.WRAP_CONTENT, LayoutParams.WRAP_CONTENT)); 
addContentView(mInfoView, new LayoutParams(LayoutParams.WRAP_CONTENT, LayoutParams.WRAP_CONTENT));

Then the problem is that the GLSurfaceView only appears after lock/unlock, and before that the screen is showing the camera and the InfoView.

We discovered that if we sleep the main thread for 4.6 seconds (or more) after executing onStart() in the activity that shows the views, the behaviour is as expected (both camera, glsurface and info views are shown, even after lock/unlock).

The thing is, we're looking for a more... elegant solution.

It seems to us that the problem is the camera taking more time than expected in Camera.open(), and so the camera View is added, the GLSurfaceView is added, and when the camera actually opens, it opens on top of the GLSurfaceView. Regarding to this, we used bringToFront() on GLSurfaceView, and got it on top of the info view, but after lock/unlock the camera still opened on top of it, leaving us with a screen with the camera preview only.

Any ideas? How can we show both SurfaceViews and the info view on top of them?

Answer 1

try this:

mLayout.addView(mRenderView);
mLayout.addView(mCustomSurfaceView);

// without this line, the camera preview cannot be displayed 
// when running activity at first time.
mCustomSurfaceView.setZOrderMediaOverlay(true);

this worked for me :)

Answer 2

Each SurfaceView's surface has a Z-depth. There are three possible depths: the default (bottom), "media overlay" (above it), and "top" (which is above everything, including the View-based UI). If you have two overlapping surfaces at the same depth, one will win, but you can't reliably define which one. You might get consistent behavior on one device only to find that it works the other way on a different device.

The composition is done with hardware overlays, when possible. Many currently-popular devices have 4 overlay planes. Once you exceed that, the composition is done with the GPU, which is going to be more expensive. If you have a status bar, navigation bar, and View UI, that's three, so you only get one SurfaceView for cheap.

For the example in the question, it would be best to combine as many things as possible onto the surface being rendered with GLES. This can include camera output on API 11+. See the "Texture from Camera" activity in Grafika for an example. (Grafika also has a demo app with three overlapping SurfaceViews; see the "multi-surface test" activity.)

If you'd like to know more about why SurfaceView behaves the way it does, see the Android System-Level Graphics doc.

Answer 3

I had the same problem. As you hint at yourself: multiple SurfaceViews don't get along with each other in that their Z order is undefined.

On my Samsung Galaxy S2 the ordering is the same as you describe (don't know how it is on other phones). The way I solved this, is checking for first time creation of the Activity in onCreate():

@Override
public void onCreate(Bundle savedInstanceState)
{
    super.onCreate(savedInstanceState);

    //...

    if ( savedInstanceState == null )
    {
        initCamView();
        initOpenGL();
    }
    else
    {
        initOpenGL();
        initCamView();
    }

    //...
}

with:

private void initOpenGL()
{
    mGLSurfaceView = new GLSurfaceView(this);
    mGLSurfaceView.setEGLConfigChooser(8, 8, 8, 8, 16, 0);
    mGLSurfaceView.getHolder().setFormat(PixelFormat.TRANSLUCENT);

    mOGLRenderer = new OGLRenderer(this);
    mGLSurfaceView.setRenderer(mOGLRenderer);
    mRL.addView(mGLSurfaceView); // mRL is the Relative Layout
}

and:

private void initCamView()
{
    mCamView = new CustomCameraView( this.getApplicationContext(),
                                     this.getWindowManager() );
    mRL.addView(mCamView); // mRL is the Relative Layout
}

Not the most elegant solution, but it's better than letting the thread sleep for 4.6 seconds.

It might also be possible to just lock the screen in a single orientation, but then you need to do a lot of ugly hacking to get the overlays rotated the right way.

Or, if you're only targeting Android 3.0 or above (API level 8), you can just show the camera in an OpenGL SurfaceTexture. http://developer.android.com/reference/android/hardware/Camera.html#setPreviewTexture(android.graphics.SurfaceTexture)