SurfaceTexture.OnFrameAvailableListener stops being called

Question

I am implementing the SurfaceTexture.OnFrameAvailableListener interface in my app so I can use the video frames as an OpenGL texture. All is setup like it should and it works perfectly however onFrameAvailable(SurfaceTexture surfaceTexture) stops being called after a few seconds, effectively and seemingly freezing the video in OpenGL as no new texturedata is being uploaded through SurfaceTexture.updateTextImage.

I am setting a flag in onFrameAvailable to do the updateTextImage call from the GL thread and only when needed. Currently I am setting the flag to true on every draw call so the video texturedata is being uploaded every frame as the onFrameAvailable check is skipped. Like this, everything runs like it should but it seems inefficient as no new texturedata needs to be uploaded if it's still the same (movie frame).

AFAIK there are no memory leaks and logcat is not showing any errors. Also, the media player is set to loop but the issue occurs before a single run has completed.

What would cause the onFrameAvailable not being called anymore after a few seconds?

Answer 1

I had this exact same problem on some devices. Found a fix and thought I would share. Basically it's what @user2254894 suggested except that since the counter could be changed by 2 different threads so its a good idea to use 2 different vars. Here is some example code:

private int             _updateTexImageCounter = 0;
private int             _updateTexImageCompare = 0;

and onFrameAvailable() is simple.

@Override
public void onFrameAvailable(SurfaceTexture surfaceTexture) 
{
    // increment every time a new frame is avail
    _updateTexImageCounter++;
}

Then in your GL update you would do something like the following...

public void update() 
{

    .... create texture... etc.
    ..
    // compare _updateTexImageCompare and _updateTexImageCounter
    if( _surfaceTexture!=null && _updateTexImageCompare != _updateTexImageCounter ) 
    {
        // loop and call updateTexImage() for each time the onFrameAvailable() method was called below.
        while(_updateTexImageCompare != _updateTexImageCounter) {
            _surfaceTexture.updateTexImage();
            _surfaceTexture.getTransformMatrix(x);

            _updateTexImageCompare++;  // increment the compare value until it's the same as _updateTexImageCounter
        }
    }


}

This worked for me. Let me know if there is a better way.

Answer 2

I've just seen a similar problem, and debugged it. I, like you, had a Boolean flag which indicated that one (or more!) frames were ready to be used.

The problem occurred when I received two camera frames between a pair of OpenGL frames (possibly because my OpenGL redraw processing was too slow). This meant that I set the Boolean flag twice. However, I then only read this frame data once, and it seems that the updateTexImage implemented some kind of queuing function.

Replacing the Boolean flag with an integer counter of pending camera frames solved the problem for me. Maybe this would work for you too?

(I suspect this is more efficient than just calling updateTexImage every frame. At least in my code, it was very rare (1-2%) for OpenGL frames to take long enough that they spanned two camera frames.)