Displaying the camera stream on a GLSurfaceView via SurfaceTexture

Question

I'm trying to display the camera stream in a GLSurfaceView via a SurfaceTexture trasmitted to OpenGL ES 2.0 shaders.

I took inspiration from this post.

The image is complete but is not correctly displayed on my tablet. The screen seems divided in 2x2 parts. The image is displayed in the upper-left part whereas the other three parts are black.

I suspect that the problem comes from my usage of the transformation matrix returned by the sequence documented here

updateTexImage();
getTransformMatrix(...);

I transmit this matrix in the vertex shader to generate the texture coordinates for the fragment shader.

vertex shader:

attribute vec3 aPosition;
uniform mat4 uMvpTransform;
// Matrix retrieved by getTransformMatrix(...);
uniform mat4 uTexMatTransform;
varying vec2 vTexCoord;

void main(void)
{
  gl_Position = uMvpTransform *vec4(aPosition.xyz, 1);
  vec4 l_tex =  uTexMatTransform*vec4(aPosition.xyz, 1);
  vTexCoord=l_tex.xy;
}

fragment shader:

#extension GL_OES_EGL_image_external : require
varying mediump vec2 vTexCoord;
uniform samplerExternalOES uSampler; 

void main(void) 
{ 
    mediump vec4 l_tex = texture2D(uSampler, vTexCoord);
    gl_FragColor=l_tex;
}

The texture is attached to the following square:

 // Image container
 GLfloat l_vertices[] = {
  -1.0f, 1.0f,  0.0f,
  -1.0f, -1.0f,  0.0f,
  1.0f,  -1.0f,  0.0f,
  1.0f,  1.0f,  0.0f };

Did anyone achieve something similar ?

Edited nov 3, 2012:

Correction of the vertex shader:

attribute vec3 aPosition;
attribute vec2 aTexCoord;
uniform mat4 uMvpTransform;
// Matrix retrieved by getTransformMatrix(...);
uniform mat4 uTexMatTransform;
varying vec2 vTexCoord;

void main(void)
{
  gl_Position = uMvpTransform *vec4(aPosition.xyz, 1);
  vec4 l_tex =  uTexMatTransform*vec4(aTexCoord.xy,0, 1);
  vTexCoord=l_tex.xy;
}

with:

// Texture
GLfloat l_texCoord[] = {
   0.0f, 1.0f,
   0.0f, 0.0f,
   1.0f,  0.0f,
   1.0f,  1.0f

};

Answer 1

I have used the SurfaceTexture succesfully to draw camera frames on a custom opengl texture without using the transformation matrix provided by android.

Just try defining your indices vertices and textures the way you would do it for a normal texture draw. Like this for example.

const GLfloat Vertices[] = {0.5, -0.5, 0,
                            0.5, 0.5, 0,
                           -0.5, 0.5, 0,
                           -0.5, -0.5, 0};

const GLubyte Indices[] = { 0, 1, 2,
                            2, 3, 0 };

const GLfloat Textures[] = { 1.,0.,
                             0.,0.,
                             0.,1.,
                             1.,1. };

You should be able to use the surfacetexture the way you use a normal texture.

In case you want to perform some sort of 3D projection, this is a good post on how to generate a proper MVP matrix. Which you could use to multiply with the position in the vertex shader.

Answer 2

Since I had the same issue, even though the question is a bit old, it might be worth it to elaborate a bit about "why use the transformation matrix".

The transformation Matrix maps the particular coordinate set used by SurfaceTexture (or more likely, by the current video source streaming into SurfaceTexture...) to the standard OpenGL texture coordinates.

The reason why it should be definitely used is that, although Startibartfast answer works in one or more particular setups, and might be easy and tempting to implement, it might produce strange visualization bugs when the same program is ran on different devices, depending for instance on the video drivers implementation of each platform.

In my case, for instance, the transform matrix just flips the content upside down (I'm using SurfaceTexture in conjunction with the MediaPlayer on Nexus 7) instead of the result pointed out by Fabien R.

The correct way to use the matrix is pointed out in Fabien R edit on nov 3, 2012, which I report with some minor embellishments (like the use of st indexes), and a small correction of int/float mismatch:

attribute   vec2 uv;
varying     vec2 vTexCoord;
uniform     mat4 transformMatrix;

vTexCoord = (transformMatrix * vec4(uv, 0.0, 1.0)).st;

Hope it helps