Android Apply Effect on Camera frame

Question

From Android 17, it provides android.media.effect package for processing images with many effects. and with the official sample 'HelloEffect', it works for image only.

Now I want apply the effects for Android camera preview frame, I used SurfaceTexture and GL_TEXTURE_EXTERNAL_OES texture to render the frame from camera.

But the problem is how to apply the effect on this special texture: GL_TEXTURE_EXTERNAL_OES, it shows errors, it seems it cannot sample from this special texture.

error:

01-06 15:46:35.425: D/libEGL(14021): loaded /vendor/lib/egl/libEGL_adreno.so
    01-06 15:46:35.435: D/libEGL(14021): loaded /vendor/lib/egl/libGLESv1_CM_adreno.so
01-06 15:46:35.435: D/libEGL(14021): loaded /vendor/lib/egl/libGLESv2_adreno.so
01-06 15:46:35.435: I/Adreno-EGL(14021): : EGL 1.4 QUALCOMM build: _msm8974_refs/tags/AU_LINUX_ANDROID_JB_3.1.2.04.02.02.125.032_CL3849330_release_AU (CL3849330)
01-06 15:46:35.435: I/Adreno-EGL(14021): OpenGL ES Shader Compiler Version: 17.01.10.SPL
01-06 15:46:35.435: I/Adreno-EGL(14021): Build Date: 10/21/13 Mon
01-06 15:46:35.435: I/Adreno-EGL(14021): Local Branch: jb_3.1.2
01-06 15:46:35.435: I/Adreno-EGL(14021): Remote Branch: quic/4da73f3b691eea7a9356efff8d609916f1975107
01-06 15:46:35.435: I/Adreno-EGL(14021): Local Patches: NONE
01-06 15:46:35.435: I/Adreno-EGL(14021): Reconstruct Branch: NOTHING
01-06 15:46:35.565: D/OpenGLRenderer(14021): Enabling debug mode 0
01-06 15:46:36.295: W/Adreno-ES20(14021): : GL_INVALID_OPERATION
01-06 15:46:36.295: E/MCA(14021): GL Error: Operation 'Binding Texture' caused GL error (0x502)
01-06 15:46:36.295: E/MCA(14021): BindInputTextures failed
01-06 15:46:36.295: E/MCA(14021): Unable to render frame
01-06 15:46:36.295: E/MCA(14021): ShaderProgram: error processing shader!
01-06 15:46:36.305: W/dalvikvm(14021): threadid=11: thread exiting with uncaught exception (group=0x419fd898)
01-06 15:46:36.345: E/AndroidRuntime(14021): FATAL EXCEPTION: GLThread 97443
01-06 15:46:36.345: E/AndroidRuntime(14021): java.lang.RuntimeException: Error executing ShaderProgram!
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.filterfw.core.ShaderProgram.process(ShaderProgram.java:123)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.filterfw.core.Program.process(Program.java:32)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.filterpacks.imageproc.LomoishFilter.process(LomoishFilter.java:200)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.filterfw.core.Filter.performProcess(Filter.java:474)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.filterfw.core.FilterFunction.execute(FilterFunction.java:80)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.filterfw.core.FilterFunction.executeWithArgList(FilterFunction.java:97)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.media.effect.SingleFilterEffect.apply(SingleFilterEffect.java:76)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at com.csz.camerafilters.activity.render.SurfaceRenderer.applyEffect(SurfaceRenderer.java:73)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at com.csz.camerafilters.activity.render.SurfaceRenderer.onDrawFrame(SurfaceRenderer.java:63)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.opengl.GLSurfaceView$GLThread.guardedRun(GLSurfaceView.java:1531)
01-06 15:46:36.345: E/AndroidRuntime(14021):    at android.opengl.GLSurfaceView$GLThread.run(GLSurfaceView.java:1248)

Answer 1

I was able to apply effects on camera frames using OpenCV SDK for Android. They have a ready sample showing that.

Answer 2

Android effect bind texture as GL_TEXTURE2D, but your texture is GL_TEXTURE_EXTERNAL_OES, that's the reason. Gen another texture of GL_TEXTURE2D, draw camera frame on it, and use this texture all the way. Just try it, this is only a theory (base on my OpenGL knowledge).