Android resize VirtualDisplay

Question

I have a few problems (actually more, but these are the more critical ones) with Android's MediaProjection API. Reading the graphics architecture won't really help, so I'm just looking to understand if I skipped something in my code flow.

Let's assume that:

1. I have a dedicated GL rendering thread, initialized, and with a GL texture generated on it. I set a default buffer size of WxH for the texture.

2. I create a SurfaceTexture using the GL texture, and a Surface for this surface texture.

3. Create a virtual display of size WxH via MediaProjection, and set its surface to the above surface.

Problem 1: everything works either perfectly (full frames come in correctly), or not (all frames are black; or only half of each frame is visible for example - same half for all frames; or some portions of the screen are repeated to other portions, sometimes even skewed).

Problem 2: while time is spent in some full-screen GL games, after a FIXED amount of time (about 4 minutes), all incoming frames are frozen (e.g. I receive "new" frames, but its actually one and the same image). Reading that with glReadPixels confirms the results - problem is, the actual display is way past that frame. The only way to force it to "recover" is to bring up the status bar or the navigation bar, which instantly starts sending me correct frames. Ofcourse, after another 4 minutes, it happens again...

Problem 3: calling resize() on the VirtualDisplay, after calling also setDefaultBUfferSize() to the GL texture, ends up in 90% of the cases to exhibit problemm #1 (either black/cut frames, artifacts of other screen areas...)

I am using the sequence of calls to updateTextureImage -> GL texture draw in the same thread, so my normal understanding is that it should never happen that I am somehow reading from a half-full GL buffer, or something... right?

I have tested this problem also by rendering the VirtualDisplay directly to the surface of a MediaCodec (no custom GL involved) - same behaviour. update Actually since MediaCodec has a fixed-size created surface, the bug cannot be reproduced because we can only resize the virtual display, but not the encoder's surface size, so this is not really a bug (but even like this it would be nice for the VirtualDisplay to resize the surface accordingly somehow).

I feel that something is leaking when closing a virtual display, or not initializing correctly between the creations of the VirtualDisplay, because it is not consistent. It may very well happen that a brand new MediaProjection screen capture permission, with a brand new virtual display, with a just-created surface texture, will end up giving me only half-cut frames... leaving me with a big poker face...

PS: all of this is happening on a Nexus 6 with Android 6.0.1.

Answer 1

I gave up and just recreated the virtual display all-together, using a new Surface of a new SurfaceTexture (but kept the GL texture). This gets rid of the flickering and half-sized frames. I also had some logcat warnings that the BufferQueue of the previously used SurfaceTexture was abandoned, and got 1 or 2 more onFrameAvailable callbacks after calling release() on the VirtualDisplay (it's probably async and should wait for the stop() callback, but it gets too complicated at that point).

And even after doing all this, something very weird still happens: sometimes the capturing will only capture MY APPLICATION. If I swipe down the status bar, or go to Home, everything turns to black (even if my app is behind the status bar window). The only solution after this is to turn the screen off/on, or to render a Camera/Camera2 preview to my SurfaceTexture, and then recreating back the VirtualDisplay. Something is definitely wrong behind the scenes regarding the texture.

It looks like VirtualDisplay.resize() is broken, and the same applies with creating a new Virtual Display using an existing SurfaceTexture.