using the pure Metal-API alongside with SceneKit or SpriteKit

Question

I have a SKView and a MTKView running in one application and everything is working well so far.

The only thing is, that both views are poorly visually integrated. They are just side by side. But I would like to have the pure metal rendering inside the SKView moving with some of the SKNodes inside. It is a kind of fast display inside the SKView.

On the metal side running quite a lot computation and rendering stuff. The SKView should provide a nice ui for the heavy calculations and minimal but very fast rendering of the pure metal part.

I already thought about using SceneKit with an overlay of a SpriteKit scene, because SCNRenderer is offering a possibility to render an own MTLCommandBuffer and MTLRenderPassDescriptor with renderAtTime.

I implemented the following SCNSceneRendererDelegate method and called my own render function, which is preparing the commandBuffer.

func renderer(renderer: SCNSceneRenderer, didRenderScene scene: SCNScene, atTime time: NSTimeInterval) {
    nodeArray.render()
}

After the commandBuffer is ready I call the renderAtTime method of my SCNRenderer. Trial and error showed me that the command buffer must committed after calling renderAtTime. If I do it before it will crash the app. If I do it not at all, it will freeze the app.

func bufferFinished(renderer:SCNRenderer, commandBuffer: MTLCommandBuffer, renderPassDescriptor: MTLRenderPassDescriptor){
    let current=CFAbsoluteTimeGetCurrent()
    renderer.renderAtTime(current, viewport: gameView.bounds, commandBuffer: commandBuffer, passDescriptor: renderPassDescriptor)
    commandBuffer.commit()
}

If I do this the app is running but no additional metal context is shown. I think so whole thing is kind of complex because of the metal part.

Is there any simple sample where pure metal is rendered in a SceneKit view or better in a SpriteKit view?

Answer 1

Metal, SpriteKit & SceneKit

SpriteKit & SceneKit working together. You can use SceneKit within SpriteKit and vice versa.

In SpriteKit you can use SK3DNode to render a SceneKit scene. Here is an Example: https://github.com/CloakedEddy/SK3DNode-example/tree/master/SKSCN%20Crossover

In SceneKit you can use the overlaySKScene property of your SCNSceneRenderer which is inherited by your SCNView. You can render your SpriteKit scenes there side by side with the SceneKit scenes.

But how to integrate metal?

Using metal with SpriteKit can be done with textures. You compute the textures with metal on the GPU and transfer the image data to a SKMutableTexture via modifyPixelDataWithBlock. This is working, but you have to copy the data which is not nice and SpriteKit is working with OpenGL behind the scenes and not with metal. So it has to create a OpenGL-Texture which is very expensive as you can see.

Another issue is, that you have to synchronize the GPU data if you working with large textures to fetch them in CPU memory. Only then you can copy the data to the SKMutableTexture. This is very inefficient since the data will go to the GPU again, but now with OpenGL.

With SceneKit you can choose if you would like to have OpenGL or Metal as underlying framework to the GPU. And you can modify the texture in the material property directly with metal. Just set the contents property of the SCNMaterialProperty. Or you can directly modify the SCNGeometrySource by metal, if you want to change the geometry of an object.