How to access EGL Image directly from Android Surface for use in MediaCodec video decoder?

Question

I'm currently writing an android app where I need to cache video-frames so that I can easily go back and forth with little to no delay.

Right now I'm letting android decode the video frame by providing a Surface to the Configure call of the MediaCodec object and calling releaseOutputBuffer with the render flag set to true.

The only way I found to access the decoded surface data (besides decoding the returned bytebuffer whose format appears to be device-dependent) is to call updateTeximage on the SurfaceTexture linked to the Surface, attaching this to the GL_TEXTURE_EXTERNAL_OES target and rendering it to a GL_TEXTURE2D target texture I created myself in order to cache it.

I would like to optimize this caching process and be able to decode the frames on a different thread. Using my current method, this means that I would have to create another EGL context for the video decoder, share the context etc...

My question is: Is it possible to access the EGL-image or native buffer data associated with the Surface without calling updateTexImage?

That way I could cache the egl image (which does not require EGL context according to EGL_ANDROID_image_native_buffer). This would also cache in YUV format which would be much more storage-efficient than the raw RGB textures I'm caching now.

Answer 1

Short answer: no.

Longer answer: the Surface encapsulates a queue of buffers. (Edit: the system is now explained in some detail here.) When you call updateTexImage(), if a new frame of data is available, the buffer at the head is dropped and the next one in the queue becomes current. Calling updateTexImage() is necessary to see successive frames; there is no mechanism for examining buffers not at the head.

A SurfaceTexture wraps an instance of GLConsumer. This consumer requires the producer (the video decoder) to generate data in a format that can be used as a "hardware texture", i.e. something the device's GL implementation can understand. It may or may not be YUV. More to the point, the consumer doesn't require that the buffer be available to "software", which means you can't assume that you can access the data directly -- you need to use GLES. (See the gralloc header for the full list of flags.)

What would be nice here is the ability to copy the buffer from the head of the BufferQueue to a separate data structure (BufferArrayList?) without doing a format conversion, but there isn't a mechanism like that at present (Android 4.3). I don't know of a better way to go about it than what you describe (shared EGL contexts, etc).

Update: My office-mate had a suggestion: use a shader to render the buffer into two textures, one for the Y and for CbCr (in GLES 3 you can use an RG texture). That keeps all the manipulation in GLES without expanding into full RGB. Internally it'll convert the MediaCodec output to RGB and grind through it twice, but that's likely cheaper than copying it out to userspace and doing it yourself on the CPU.