Using Pixel Buffer Objects (PBO's) on Android

Question

On Android, I'm trying to perform some OpenGL processing on camera frames, show those frames in the camera preview and then encode the frames in a video file. I'm trying to do this with OpenGL, using the GLSurfaceView and GLSurfaceView.Renderer and with FFMPEG for video encoding.

I've successfully processed the image frames using a shader. Now I need to encode the processed frames to video. The GLSurfaceView.Renderer provides the onDrawFrame(GL10 ..) method. It's in this method that I'm attempting to read the image frames using just glReadPixels() and then place the frames on a queue for encoding to video. On it's own, glReadPixels() is much too slow - my frame rate is in the single digits. I'm attempting to speed this up using Pixel Buffer Objects. This is not working. After plugging in the pbo, the frame rate is unchanged. This is my first time using OpenGL and I do not know where to begin looking for the problem. Am I doing this right? Can anyone give me some direction? Thanks in advance.

public class MainRenderer implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener {

.
.

    public void onDrawFrame ( GL10 gl10 ) {

        //Create a buffer to hold the image frame     
        ByteBuffer byte_buffer = ByteBuffer.allocateDirect(this.width * this.height * 4);
        byte_buffer.order(ByteOrder.nativeOrder());

        //Generate a pointer to the frame buffers
        IntBuffer image_buffers = IntBuffer.allocate(1); 
        GLES20.glGenBuffers(1, image_buffers);

        //Create the buffer
        GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, image_buffers.get(0));
        GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, byte_buffer.limit(), byte_buffer, GLES20.GL_STATIC_DRAW);
        GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, image_buffers.get(0));

        //Read the pixel data into the buffer
        gl10.glReadPixels(0, 0, this.width, this.height, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, byte_buffer);

        //encode the frame to video
        enQueueForEncoding(byte_buffer);

        //unbind the buffer
        GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
    }

.
.

}

Answer 1

I have never tried something like that before (opengl+video enconding) but I can tell you that reading from device memory is SLOW. Try double buffering, this may help since the GPU can keep rendering to the second buffer while the DMA controller reads back stuff.

Load a profiler (check your devices' GPU vendor), this may give you some idea. Another thing that may help is setting internal pbuffer format to something else, try lower numbers and dropping a channel (alpha).

EDIT: If you feel like that, you can encode the video at the GPU, that's going to boost, memory and processing wise, your application.

Answer 2

As I remember glBufferData() is not mapping your internal buffer onto GPU memory, it just copies data from your memory into the buffer (initializes).

To get access to the memory, which is allocated by glBufferData(), you should use glMapBufferRange(). That function returns a Java Buffer object which you can read.