Android OpenGL combination of SurfaceTexture (external image) and ordinary texture

Question

I would like to mix camera preview SurfaceTexture with some overlay texture. I am using these shaders for processing:

private final String vss = "attribute vec2 vPosition;\n"
        + "attribute vec2 vTexCoord;\n"
        + "varying vec2 texCoord;\n"
        + "void main() {\n" 
        + "  texCoord = vTexCoord;\n"
        + "  gl_Position = vec4 ( vPosition.x, vPosition.y, 0.0, 1.0 );\n"
        + "}";

private final String fss = "#extension GL_OES_EGL_image_external : require\n"
        + "precision mediump float;\n"
        + "uniform samplerExternalOES sTexture;\n"
        + "uniform sampler2D filterTexture;\n"
        + "varying vec2 texCoord;\n"
        + "void main() {\n"
        +"  vec4 t_camera = texture2D(sTexture,texCoord);\n"
        //+"  vec4 t_overlayer = texture2D(filterTexture, texCoord);\n" 
        //+ "  gl_FragColor = t_overlayer;\n" + "}";
        + "  gl_FragColor = t_camera;\n" + "}";

My goal is to mix t_camera and t_overlayer. When I show t_camera or t_overlayer separately, it works (showing camera preview or texture). But when I uncomment t_overlayer, then t_camera becomes black (somehow badly sampled). My overlayer texture is 512x512 and CLAMPT_TO_EDGE. This problem occurs only for example on: Android Emulator, HTC Evo 3D. But on SGS3, HTC One X, it works just fine.

What is wrong? Is it Evo 3D missing some extension or what?

Answer 1

In order to convert an external texture (non-GPU) to a regular internal one, you have to

• First create an external texture (same as creating a regular texture, but replacing all GLES20.GL_TEXTURE_2D with GLES11Ext.GL_TEXTURE_EXTERNAL_OES).
• Create an internal/regular texture to write to
• Wrap a SurfaceTexture with the external texture.
• Read the content into that external texture (if from a camera, a file, etc.)
• Override onFrameAvailable of that SurfaceTexture and do the conversion here, supplying both the external texture to read from, and the internal texture to write to.
• You might need to call getTransformMatrix for corrections in coordinates (usually flip y axis) and supply it as well. Sometimes not...

here's some example shaders:

Vertex-shdaer -

uniform mat4 transform; // might be needed, and might not
uniform mat4 modelview;
uniform mat4 projection;
attribute vec2 position;

varying vec2 vTexcoord;

void main() {
    gl_Position = projection * modelview * vec4(position.xy, 0.0, 1.0);
    // texture takes points in [0,1], while position is [-1,1];
    vec4 newpos = (gl_Position + 1.0) * 0.5;
    vTexcoord = (transform * newpos).xy ;
}

Fragment shader -

#extension GL_OES_EGL_image_external : require

precision mediump float;

uniform samplerExternalOES sTexture;
varying vec2 vTexcoord;

void main() {
    gl_FragColor = texture2D(sTexture, vTexcoord);
}

Answer 2

I imagine you have this problem because you are not setting the correct texture id on your code. This is a tipical error on assumptions which seem logical, but is actually not defined so on the documentation. If you check the documentation of this extension you see the following (edited) TEXT:

Each TEXTURE_EXTERNAL_OES texture object may require up to 3 texture image units for each texture unit to which it is bound. When is set to TEXTURE_EXTERNAL_OES this value will be between 1 and 3 (inclusive). For other valid texture targets this value will always be 1. Note that, when a TEXTURE_EXTERNAL_OES texture object is bound, the number of texture image units required by a single texture unit may be 1, 2, or 3, while for other texture objects each texture unit requires exactly 1 texture image unit.

This means that at leas one additional will work, provided that you use id 0 for it. In your case:

GLES20.glUniform1i(sTextureHandle, 1);
GLES20.glActiveTexture(GLES20.GL_TEXTURE1);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
        sTextureId);

For your 2D texture:

GLES20.glUniform1i(filterTextureHandle, 0);
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, filterTextureID);

I'm sure this will workout for you.

Answer 3

I got the same issue on my Nexus 7 and it drove me crazy. Accessing either a samplerExternalOES or a sampler2D worked totally fine, but accessing them both in the same shader gave unexpected results. Sometimes the output would be black. Sometimes the output of one of the lookup would have bad quantization artifacts. The behaviour would also vary depending on the texture unit the samplers where bound to. I did check every opengl error and validateProgram results.

Eventually, what worked was to use a separate shader to simply access the camera output and render that into a texture. Then the resulting texture can be accessed through a regular sampler2D and everything works exactly as expected. I suspect there's a bug somewhere related to samplerExternalOES.

Answer 4

The method on the above save lots of my time. Thank you guru:

GLES20.glUniform1i(sTextureHandle, 1);
GLES20.glActiveTexture(GLES20.GL_TEXTURE1);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
    sTextureId);

For your 2D texture:

GLES20.glUniform1i(filterTextureHandle, 0);
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, filterTextureID);

Change the texture index is a good way to solve this.