Match ARSCNView transforms for custom Scenekit rendering?

Question

TL:DR

I am trying to render an SCNRenderer in a MTKView so I have handles on the color / depth buffers - and I am trying to match the transforms that ARSCNView does via ARKit so my SCNRenderer behaves like an ARSCNView.

Does anyone have an example of matching ARSCNViews world and face tracking configurations via the ARFrame/ARCamera and info provided by ARKit - and properly modifying the transforms of SCNSCamera appropriately?

Longer

For various reasons (not having access to a MTLPassDescriptor for ARSCNView / SCNView and SCNTechnique having its own weird limitations) I'm trying to re-implement some of ARSCNView via MTKView + ARKit + SCNRenderer myself, which allows me to have more control over the rendering pipeline. I am basing my code off of Apples ARKit + Metal example.

I'd like to match what ARKit is doing - but when I attempt to use the ARFrame / ARCamera information to modify my SCNScene and SCNCamera as I show below, my scene moves, but not with the camera in the same way that either ARSCNView or the ARKit with Metal demo does.

I'm at a loss as to what is incorrectly being passed to my scene from ARKit.

My Scene:

self.sceneKitRenderer = SCNRenderer(device: self.device, options: nil)

self.sceneKitRenderer?.delegate = self as SCNSceneRendererDelegate


let scene = SCNScene()
scene.background.contents = nil

let sphere = SCNSphere(radius: 1.0)
sphere.firstMaterial?.diffuse.contents = UIColor.red
sphere.firstMaterial?.isDoubleSided = true
let sphereNode = SCNNode(geometry: sphere)
sphereNode.position = SCNVector3(0, 0.0, -5)
scene.rootNode.addChildNode(sphereNode)


let camera = SCNCamera()
let cameraNode = SCNNode()
cameraNode.camera = camera
cameraNode.position = SCNVector3(x: 0, y: 0, z: 10)
cameraNode.look(at: scene.rootNode.worldPosition)
scene.rootNode.addChildNode(cameraNode)

self.sceneKitRenderer?.scene = scene
self.sceneKitRenderer?.pointOfView = cameraNode

I am updating my Scene's point of view and its projection matrix when I receive an updated ARFrame like so:

// Match clipping
self.sceneKitRenderer?.pointOfView?.camera?.zNear = 0.001
self.sceneKitRenderer?.pointOfView?.camera?.zFar = 1000

// Match projection
let projection = SCNMatrix4( frame.camera.projectionMatrix(for: .landscapeRight, viewportSize: viewportSize, zNear: 0.001, zFar: 1000))
self.sceneKitRenderer?.pointOfView?.camera?.projectionTransform = projection

// Match transform
self.sceneKitRenderer?.pointOfView?.simdTransform = frame.camera.transform

While this appears to do something, it isn't matching ARKits output.

Questions:

Is assigning the simdTransform or the SCNRenderer's pointOfView from the camera's translation correct?

Do I need to do something to my scene to set up the proper coordinate system (ie, a root node scale?)

Any guidance is appreciated!

Answer 1

The code referenced in the question for matching the ARCamera with the SCNScene camera won't work for different orientations, as the transform will differ depending on the orientation. There is a function on ARCamera to get the view matrix for a particular orientation(the view matrix being the inverse of the camera transform). So the following can be used to correctly take orientation in to account:

// Match clipping
self.sceneKitRenderer?.pointOfView?.camera?.zNear = 0.001
self.sceneKitRenderer?.pointOfView?.camera?.zFar = 1000

// Match projection
let projection = SCNMatrix4( frame.camera.projectionMatrix(for: .portrait, viewportSize: viewportSize, zNear: 0.001, zFar: 1000))
self.sceneKitRenderer?.pointOfView?.camera?.projectionTransform = projection

// Match transform
self.sceneKitRenderer?.pointOfView?.simdTransform = currentFrame.camera.viewMatrix(for: .portrait).inverse

Answer 2

I did two things to get this working. First you need to get the camera node from ARKit's scene and use this to set the camera node's transform and the camera's projectionTransform in your other scene:

// get the first node that has a camera attached
if let arkitCameraNode = arkitScene.rootNode.childNodes(passingTest: { (node, stop) -> Bool in
     return node.camera != nil
}).first {
     // SCNNode
     otherSceneCameraNode.transform = arkitCameraNode.transform
     // SCNCamera
     otherSceneCamera.projectionTransform = arkitCameraNode.camera!.projectionTransform
}

After this you need to sync the cloned nodes in your other scene by getting its corresponding ARAnchor transform:

if let anchor = arkitSceneView.anchor(for: nodeFromARKitScene) {
     clonedNodeInOtherScene.simdTransform = anchor.transform
}

Hope this helps!