MediaCodec with Surface input: Producing chunked output

Question

I'm trying to produce short sequential mp4 files from CameraPreview data via MediaCodec.createInputSurface(). However, recreating the MediaCodec and it's associated Surface requires stopping the Camera to allow another call to mCamera.setPreviewTexture(...). This delay results in an unacceptable amount of dropped frames.

Therefore I need to generate the CODEC_CONFIG and END_OF_STREAM data periodically without recreating the input Surface, and thus having to call mCamera.setPreviewTexture(...). Is this possible assuming the MediaFormat is unchanged?

(I'm adapting fadden's CameraToMpegTest example. My complete code is here)

Unsuccessful attempts:

Calling MediaCodec.signalEndOfInputStream(), draining the MediaCodec, and then calling MediaCodec.flush() between chunks produces an IllegalStateException on the 2nd call to MediaCodec.signalEndOfInputStream().

Calling MediaCodec.signalEndOfInputStream(), draining the MediaCodec, and then calling MediaCodec.stop(); MediaCodec.configure(...), MediaCodec.start() between chunks without again calling MediaCodec.createInputSurface() produces the following error:

    09-30 13:12:49.889  17638-17719/x.xx.xxxx E/Surface﹕ queueBuffer: error queuing buffer to SurfaceTexture, -19
09-30 13:12:49.889  17638-17719/x.xx.xxxx E/IMGSRV﹕ :0: UnlockPostBuffer: Failed to queue buffer 0x592e1e70
09-30 13:12:49.889  17638-17719/x.xx.xxxx E/CameraToMpegTest﹕ Encoding loop exception!
09-30 13:12:49.889  17638-17719/x.xx.xxxx W/System.err﹕ java.lang.RuntimeException: eglSwapBuffers: EGL error: 0x300b
09-30 13:12:49.896  17638-17719/x.xx.xxxx W/System.err﹕ at x.xx.xxxx.ChunkedHWRecorder$CodecInputSurface.checkEglError(ChunkedHWRecorder.java:731)
09-30 13:12:49.896  17638-17719/x.xx.xxxx W/System.err﹕ at x.xx.xxxx.ChunkedHWRecorder$CodecInputSurface.swapBuffers(ChunkedHWRecorder.java:713)
09-30 13:12:49.896  17638-17719/x.xx.xxxx W/System.err﹕ at x.xx.xxxx.ChunkedHWRecorder.startRecording(ChunkedHWRecorder.java:164)
09-30 13:12:49.896  17638-17719/x.xx.xxxx W/System.err﹕ at x.xx.xxxx.HWRecorderActivity$CameraToMpegWrapper.run(HWRecorderActivity.java:76)
09-30 13:12:49.896  17638-17719/x.xx.xxxx W/System.err﹕ at java.lang.Thread.run(Thread.java:841)

Solved Thanks fadden. The complete solution source is here.

Answer 1

The signalEndOfInputStream() call updates the state of various layers in the MediaCodec stack. You can get some sense of what operations are valid from the comments above the tests in MediaCodecTest, but by and large the behavior of MediaCodec is simply not defined for "unusual" uses.

So you have to look at the code. The lifetime of the input surface is tied to that of the OMXNodeInstance; it's represented by GraphicBufferSource. Once you signal EOS, the GraphicBufferSource will ignore additional frames (see line 426). There's no way to reset the EOS flag without tearing down the GraphicBufferSource, but when you do that it disconnects the buffer queue that underlies the Surface.

So I don't think you're going to be able to stop/restart the MediaCodec and continue to use the Surface.

However... you shouldn't need to. CameraToMpegTest routes the camera preview to a SurfaceTexture, and then renders the texture onto the encoder's input surface with GLES. The SurfaceTexture is decoupled from the encoder and shouldn't need to change. I think what needs to change is CodecInputSurface, which calls eglCreateWindowSurface() with the Surface from the MediaCodec to tell GLES where to draw. If you add a new "update Surface" API there (destroy old EGLSurface, create new EGLSurface, eglMakeCurrent), and call it whenever you spin up a new MediaCodec, I think it'll all just work.

Update to address comments:

It's important that you only change the EGLSurface. The checkAndUpdateEglStateLocked() function in GLConsumer.cpp checks to make sure the EGLDisplay and EGLContext don't change once they've been set. You can't call release()/eglSetup() in CodecInputSurface because it changes the EGLContext. You just want to destroy and recreate the EGLSurface.