ARKit – Replicating robot character in Motion Capture RealityKit

Question

I'm trying to make a 3d model like robot provided by Apple in Motion Capture example (shown at WWDC 2019) which can mimic me in motion capture ARKit 3.0 by replacing robot character given by Apple.

Desired Solution:

• Is there any special software which Apple used to create robot.usdz file? If yes, then please provide details for it?

• How can we convert formats like .glb/.gltf/.obj/.dae file to .usdz using Apple’s Python based tool without affecting it’s scene graph?

• How can we edit the scene graph of a .usdz file in Xcode and successfully save the changes in a .usdz file?

Answer 1

Working ARKit's MoCap solution

• Create a character in Autodesk Maya 2020 | 2022 | 2023
• Download Apple's model biped skeleton and import it into Maya with Maya USD plugin
• Appropriately rig a Apple's skeleton for a Motion Capture
• Bind skeleton to a mesh using Skin–Bind Skin
• Select skeleton and character mesh in Outliner
• Export a model as USD or FBX file format

Then download USDZ Tools for Xcode 12 and FBX Python SDK.

Here's my POST where you'll find instructions how to create a .zshrc file in macOS.

Test MoCap-ed model:

import RealityKit
import ARKit

class ViewController: UIViewController, ARSessionDelegate {

    @IBOutlet var arView: ARView!
    
    var character: Entity?
    let characterAnchor = AnchorEntity()
    
    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)

        arView.session.delegate = self

        guard ARBodyTrackingConfiguration.isSupported
        else { fatalError("MoCap is available on A12 & later") }
        
        let config = ARBodyTrackingConfiguration()
        arView.session.run(config)
        arView.scene.addAnchor(characterAnchor)
        
        character = try? Entity.load(named: "character")
    }
    
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        
        for anchor in anchors {
            
            guard let bodyAnchor = anchor as? ARBodyAnchor
            else { continue }

            let bodyPosition = simd_make_float3(bodyAnchor.transform.columns.3)
            characterAnchor.position = bodyPosition
            characterAnchor.orientation = Transform(matrix: bodyAnchor.transform).rotation

            if let character = character, character.parent == nil {

                characterAnchor.addChild(character)
                characterAnchor.scale = [0.02, 0.02, 0.02]
            }
        }
    }
}