Holding onto a MTLTexture from a CVImageBuffer causes stuttering

Question

I'm creating a MTLTexture from CVImageBuffers (from camera and players) using CVMetalTextureCacheCreateTextureFromImage to get a CVMetalTexture and then CVMetalTextureGetTexture to get the MTLTexture.

The problem I'm seeing is that when I later render the texture using Metal, I occasionally see video frames rendered out of order (visually it stutters back and forth in time), presumably because CoreVideo is modifying the underlying CVImageBuffer storage and the MTLTexture is just pointing there.

Is there any way to make CoreVideo not touch that buffer and use another one from its pool until I release the MTLTexture object?

My current workaround is blitting the texture using a MTLBlitCommandEncoder but since I just need to hold on to the texture for ~30 milliseconds that seems unnecessary.

Answer 1

I recently ran into this exact same issue. The problem is that the MTLTexture is not valid unless it's owning CVMetalTextureRef is still alive. You must keep a reference to the CVMetalTextureRef the entire time you're using the MTLTexture (all the way until the end of the current rendering cycle).

Answer 2

I ran into the same problem, but having an extra reference to CVMetalTexture object did NOT solve this problem in my case.

As far as I can tell, it happens only when I receive a new frame from the camera before my metal code completes processing the previous frame.

It seems that CVMetalTextureCacheCreateTextureFromImage simply creates a texture on top of the pixel buffer that the camera is feeding data into it. Therefore, accessing it from Metal code asynchronously is cause some issues.

I have decided to create a copy of MTLTexture (which is also asynchronous but is fast enough).

Here is description of CVMetalTextureCacheCreateTextureFromImage()

"This function creates or returns a cached CoreVideo Metal texture buffer mapped to an image buffer according to the specified, creating a live binding between a device-based image buffer and a MTLTexture object.",