Get CVPixelBuffer (camera frame along with ar models) in a performant manner

Question

I need to get a CVPixelBuffer containing the camera frame along with the AR models that I've placed, at a rate of 30+ fps, preferably with low energy & CPU impact.

The capturedImage from the frame in session(_:didUpdate:) doesn't contain the AR models.

I've tried using sceneView.snapshot() to get UIImage which I then convert to CVPixelBuffer. This has a noticeably high energy impact.

Finally, I've also tried creating a SCNRenderer which I then use to get the UIImage via snapshot(atTime:with:antialiasingMode:) and later convert to CVPixelBuffer. This has a slightly lesser CPU and energy impact. But this only works for portrait mode, the generated UIImage is incorrect in landscape mode.

Is there a way to get CVPixelBuffer in a less CPU and energy intensive manner(at least when compared to the above)?

Answer 1

You can create your own CVPixelBuffer, then get a MTLTexture from it and finally have SCNRenderer render into that texture. This won't involve the CPU cost of generating a UIImage from the snapshot API.

The key part here is to use a CVMetalTextureCache to obtain a Metal texture from a pixel buffer.