Rendering camera into multiple surfaces - on and off screen

Question

I want to render the camera output into a view and once in a while save the camera output frame to a file, with the constraint being - the saved frame should be the same resolution as the camera is configured, while the view is smaller than the camera output (maintaining the aspect ratio).

Based on the ContinuousCaptureActivity example in grafika, I thought the best approach would be to send the camera to a SurfaceTexture and generally rendering the output and downscaling it into a SurfaceView, and when needed, render the full frame into a different Surface that has no view, in order to retrieve a byte buffer from it in parallel to the regular SurfaceView rendering.

The example is very similar to my situation - the preview is rendered to a view of smaller size and can be recorded and saved at the full resolution via a VideoEncoder.

I replaced the VideoEncoder logic with my own and got stuck trying to provide a Surface, like the encoder does, for the full resolution rendering. How do I create such a Surface? Am I approaching this correctly?

---

Some code ideas based on the example:

---

Inside the surfaceCreated(SurfaceHolder holder) method (line 350):

@Override   // SurfaceHolder.Callback
public void surfaceCreated(SurfaceHolder holder) {
    Log.d(TAG, "surfaceCreated holder=" + holder);

    mEglCore = new EglCore(null, EglCore.FLAG_RECORDABLE);
    mDisplaySurface = new WindowSurface(mEglCore, holder.getSurface(), false);
    mDisplaySurface.makeCurrent();

    mFullFrameBlit = new FullFrameRect(
            new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT));
    mTextureId = mFullFrameBlit.createTextureObject();
    mCameraTexture = new SurfaceTexture(mTextureId);
    mCameraTexture.setOnFrameAvailableListener(this);

    Log.d(TAG, "starting camera preview");
    try {
        mCamera.setPreviewTexture(mCameraTexture);
    } catch (IOException ioe) {
        throw new RuntimeException(ioe);
    }
    mCamera.startPreview();


    // *** MY EDIT START ***

    // Encoder creation no longer needed
    //  try {
    //    mCircEncoder = new CircularEncoder(VIDEO_WIDTH, VIDEO_HEIGHT, 6000000,
    //            mCameraPreviewThousandFps / 1000, 7, mHandler);
    //  } catch (IOException ioe) {
    //      throw new RuntimeException(ioe);
    //  }

    mEncoderSurface = new WindowSurface(mEglCore, mCameraTexture); // <-- Crashes with EGL error 0x3003

    // *** MY EDIT END ***

    updateControls();
}

The drawFrame() method (line 420):

private void drawFrame() {
    //Log.d(TAG, "drawFrame");
    if (mEglCore == null) {
        Log.d(TAG, "Skipping drawFrame after shutdown");
        return;
    }

    // Latch the next frame from the camera.
    mDisplaySurface.makeCurrent();
    mCameraTexture.updateTexImage();
    mCameraTexture.getTransformMatrix(mTmpMatrix);

    // Fill the SurfaceView with it.
    SurfaceView sv = (SurfaceView) findViewById(R.id.continuousCapture_surfaceView);
    int viewWidth = sv.getWidth();
    int viewHeight = sv.getHeight();
    GLES20.glViewport(0, 0, viewWidth, viewHeight);
    mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
    mDisplaySurface.swapBuffers();

    // *** MY EDIT START ***

    // Send it to the video encoder.
    if (someCondition) {
        mEncoderSurface.makeCurrent();
        GLES20.glViewport(0, 0, VIDEO_WIDTH, VIDEO_HEIGHT);
        mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
        mEncoderSurface.swapBuffers();
        try {
            mEncoderSurface.saveFrame(new File(getExternalFilesDir(null), String.valueOf(System.currentTimeMillis()) + ".png"));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    // *** MY EDIT END ***

}

Answer 1

You're on the right track. The SurfaceTexture just does a quick bit of wrapping around the original YUV frame from the camera, so the "external" texture is the original image, with no changes. You can't read the pixels straight out of an external texture, so you have to render it somewhere first.

The easiest way to do this is to create an off-screen pbuffer surface. Grafika's gles/OffscreenSurface class does exactly this (with a call to eglCreatePbufferSurface()). Make that EGLSurface current, render the texture onto a FullFrameRect, then read the framebuffer with glReadPixels() (see EglSurfaceBase#saveFrame() for code). Don't call eglSwapBuffers().

Note that you're not creating an Android Surface for the output, just an EGLSurface. (They're different.)