Buffering Surface input to MediaCodec

Question

It's been demonstrated how to feed MediaCodec with Surface input like the CameraPreview, but are there practical ways of buffering this input before submission to MediaCodec?

In my experiments, a Galaxy Nexus experiences unacceptable hiccups in producing audio / video streams using the direct, synchronous encoding method in CameraToMpegTest.java

When using MediaCodec with byte[] or ByteBuffer input, we can submit unencoded data to a ExecutorService or similar queue for processing to ensure no frames are dropped, even if the device experiences spikes in CPU usage out of our application's control. However, due to the requirement of performing color format conversion between Android's Camera and MediaCodec this method is unrealistic for high resolution, live video.

Thoughts:

1. Is there a way to feed the NativePixmapType created with EGL14.eglCopyBuffers(EGLDisplay d, EGLSurface s, NativePixmapType p) to MediaCodec?

2. Can anyone from Android comment on whether harmonizing ByteBuffer formats between the Camera and MediaCodec is on the roadmap?

Answer 1

You really don't want to copy the data at all. Allocating storage for and copying a large chunk of data can take long enough to kill your frame rate. This generally rules out byte[] and ByteBuffer[] solutions, even if you didn't have to do a U/V plane swap.

The most efficient way to move data through the system is with a Surface. The trick is that a Surface isn't a buffer, it's an interface to a queue of buffers. The buffers are passed around by reference; when you unlockCanvasAndPost() you're actually placing the current buffer onto a queue for the consumer, which is often in a different process.

There is no public mechanism for creating a new buffer and adding it to the set used by the queue, or for extracting buffers from the queue, so you can't implement a DIY buffering scheme on the side. There's no public interface to change the number of buffers in the pool.

It'd be useful to know what it is that's causing the hiccups. The Android tool for analyzing such issues is systrace, available in Android 4.1+ (docs, example, bigflake example). If you can identify the source of the CPU load, or determine that it's not CPU but rather some bit of code getting tangled up, you'll likely have a solution that's much easier than adding more buffers to Surface.