ARKIT: Move Object with PanGesture (the right way)

Question

I've been reading plenty of StackOverflow answers on how to move an object by dragging it across the screen. Some use hit tests against .featurePoints some use the gesture translation or just keeping track of the lastPosition of the object. But honestly.. none work the way everyone is expecting it to work.

Hit testing against .featurePoints just makes the object jump all around, because you dont always hit a featurepoint when dragging your finger. I dont understand why everyone keeps suggesting this.

Solutions like this one work: Dragging SCNNode in ARKit Using SceneKit

But the object doesnt really follow your finger, and the moment you take a few steps or change the angle of the object or the camera.. and try to move the object.. the x,z are all inverted.. and makes total sense to do that.

I really want to move objects as good as the Apple Demo, but I look at the code from Apple... and is insanely weird and overcomplicated I cant even understand a bit. Their technique to move the object so beautifly is not even close to what everyone propose online. https://developer.apple.com/documentation/arkit/handling_3d_interaction_and_ui_controls_in_augmented_reality

There's gotta be a simpler way to do it.

Answer 1

Short answer: To get this nice and fluent dragging effect like in the Apple demo project, you will have to do it like in the Apple demo project (Handling 3D Interaction). On the other side I agree with you, that the code might be confusing if you look at it for the first time. It is not easy at all to calculate the correct movement for an object placed on a floor plane - always and from every location or viewing angle. It’s a complex code construct, that is doing this superb dragging effect. Apple did a great job to achieve this, but didn’t make it too easy for us.

Full Answer: Striping down the AR Interaction template for your needy results in a nightmare - but should work too if you invest enough time. If you prefer to begin from scratch, basically start using a common swift ARKit/SceneKit Xcode template (the one containing the space ship).

You will also require the entire AR Interaction Template Project from Apple. (The link is included in the SO question) At the End you should be able to drag something called VirtualObject, which is in fact a special SCNNode. In Addition you will have a nice Focus Square, that can be useful for whatever purpose - like initially placing objects or adding a floor, or a wall. (Some code for the dragging effect and the focus square usage are kind of merged or linked together - doing it without the focus square will actually be more complicated)

Get started: Copy the following files from the AR Interaction template to your empty project:

• Utilities.swift (usually I name this file Extensions.swift, it contains some basic extensions that are required)
• FocusSquare.swift
• FocusSquareSegment.swift
• ThresholdPanGesture.swift
• VirtualObject.swift
• VirtualObjectLoader.swift
• VirtualObjectARView.swift

Add the UIGestureRecognizerDelegate to the ViewController class definition like so:

class ViewController: UIViewController, ARSCNViewDelegate, UIGestureRecognizerDelegate {

Add this code to your ViewController.swift, in the definitions section, right before viewDidLoad:

// MARK: for the Focus Square
// SUPER IMPORTANT: the screenCenter must be defined this way
var focusSquare = FocusSquare()
var screenCenter: CGPoint {
    let bounds = sceneView.bounds
    return CGPoint(x: bounds.midX, y: bounds.midY)
}
var isFocusSquareEnabled : Bool = true


// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
/// The tracked screen position used to update the `trackedObject`'s position in `updateObjectToCurrentTrackingPosition()`.
private var currentTrackingPosition: CGPoint?

/**
 The object that has been most recently intereacted with.
 The `selectedObject` can be moved at any time with the tap gesture.
 */
var selectedObject: VirtualObject?

/// The object that is tracked for use by the pan and rotation gestures.
private var trackedObject: VirtualObject? {
    didSet {
        guard trackedObject != nil else { return }
        selectedObject = trackedObject
    }
}

/// Developer setting to translate assuming the detected plane extends infinitely.
let translateAssumingInfinitePlane = true
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***

In viewDidLoad, before you setup the scene add this code:

// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
let panGesture = ThresholdPanGesture(target: self, action: #selector(didPan(_:)))
panGesture.delegate = self

// Add gestures to the `sceneView`.
sceneView.addGestureRecognizer(panGesture)
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***

At the very end of your ViewController.swift add this code:

// MARK: - Pan Gesture Block
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
@objc
func didPan(_ gesture: ThresholdPanGesture) {
    switch gesture.state {
    case .began:
        // Check for interaction with a new object.
        if let object = objectInteracting(with: gesture, in: sceneView) {
            trackedObject = object // as? VirtualObject
        }

    case .changed where gesture.isThresholdExceeded:
        guard let object = trackedObject else { return }
        let translation = gesture.translation(in: sceneView)

        let currentPosition = currentTrackingPosition ?? CGPoint(sceneView.projectPoint(object.position))

        // The `currentTrackingPosition` is used to update the `selectedObject` in `updateObjectToCurrentTrackingPosition()`.
        currentTrackingPosition = CGPoint(x: currentPosition.x + translation.x, y: currentPosition.y + translation.y)

        gesture.setTranslation(.zero, in: sceneView)

    case .changed:
        // Ignore changes to the pan gesture until the threshold for displacment has been exceeded.
        break

    case .ended:
        // Update the object's anchor when the gesture ended.
        guard let existingTrackedObject = trackedObject else { break }
        addOrUpdateAnchor(for: existingTrackedObject)
        fallthrough

    default:
        // Clear the current position tracking.
        currentTrackingPosition = nil
        trackedObject = nil
    }
}

// - MARK: Object anchors
/// - Tag: AddOrUpdateAnchor
func addOrUpdateAnchor(for object: VirtualObject) {
    // If the anchor is not nil, remove it from the session.
    if let anchor = object.anchor {
        sceneView.session.remove(anchor: anchor)
    }

    // Create a new anchor with the object's current transform and add it to the session
    let newAnchor = ARAnchor(transform: object.simdWorldTransform)
    object.anchor = newAnchor
    sceneView.session.add(anchor: newAnchor)
}


private func objectInteracting(with gesture: UIGestureRecognizer, in view: ARSCNView) -> VirtualObject? {
    for index in 0..<gesture.numberOfTouches {
        let touchLocation = gesture.location(ofTouch: index, in: view)

        // Look for an object directly under the `touchLocation`.
        if let object = virtualObject(at: touchLocation) {
            return object
        }
    }

    // As a last resort look for an object under the center of the touches.
    // return virtualObject(at: gesture.center(in: view))
    return virtualObject(at: (gesture.view?.center)!)
}


/// Hit tests against the `sceneView` to find an object at the provided point.
func virtualObject(at point: CGPoint) -> VirtualObject? {

    // let hitTestOptions: [SCNHitTestOption: Any] = [.boundingBoxOnly: true]
    let hitTestResults = sceneView.hitTest(point, options: [SCNHitTestOption.categoryBitMask: 0b00000010, SCNHitTestOption.searchMode: SCNHitTestSearchMode.any.rawValue as NSNumber])
    // let hitTestOptions: [SCNHitTestOption: Any] = [.boundingBoxOnly: true]
    // let hitTestResults = sceneView.hitTest(point, options: hitTestOptions)

    return hitTestResults.lazy.compactMap { result in
        return VirtualObject.existingObjectContainingNode(result.node)
        }.first
}

/**
 If a drag gesture is in progress, update the tracked object's position by
 converting the 2D touch location on screen (`currentTrackingPosition`) to
 3D world space.
 This method is called per frame (via `SCNSceneRendererDelegate` callbacks),
 allowing drag gestures to move virtual objects regardless of whether one
 drags a finger across the screen or moves the device through space.
 - Tag: updateObjectToCurrentTrackingPosition
 */
@objc
func updateObjectToCurrentTrackingPosition() {
    guard let object = trackedObject, let position = currentTrackingPosition else { return }
    translate(object, basedOn: position, infinitePlane: translateAssumingInfinitePlane, allowAnimation: true)
}

/// - Tag: DragVirtualObject
func translate(_ object: VirtualObject, basedOn screenPos: CGPoint, infinitePlane: Bool, allowAnimation: Bool) {
    guard let cameraTransform = sceneView.session.currentFrame?.camera.transform,
        let result = smartHitTest(screenPos,
                                  infinitePlane: infinitePlane,
                                  objectPosition: object.simdWorldPosition,
                                  allowedAlignments: [ARPlaneAnchor.Alignment.horizontal]) else { return }

    let planeAlignment: ARPlaneAnchor.Alignment
    if let planeAnchor = result.anchor as? ARPlaneAnchor {
        planeAlignment = planeAnchor.alignment
    } else if result.type == .estimatedHorizontalPlane {
        planeAlignment = .horizontal
    } else if result.type == .estimatedVerticalPlane {
        planeAlignment = .vertical
    } else {
        return
    }

    /*
     Plane hit test results are generally smooth. If we did *not* hit a plane,
     smooth the movement to prevent large jumps.
     */
    let transform = result.worldTransform
    let isOnPlane = result.anchor is ARPlaneAnchor
    object.setTransform(transform,
                        relativeTo: cameraTransform,
                        smoothMovement: !isOnPlane,
                        alignment: planeAlignment,
                        allowAnimation: allowAnimation)
}
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***

Add some Focus Square Code

// MARK: - Focus Square (code by Apple, some by me)
func updateFocusSquare(isObjectVisible: Bool) {
    if isObjectVisible {
        focusSquare.hide()
    } else {
        focusSquare.unhide()
    }

    // Perform hit testing only when ARKit tracking is in a good state.
    if let camera = sceneView.session.currentFrame?.camera, case .normal = camera.trackingState,
        let result = smartHitTest(screenCenter) {
        DispatchQueue.main.async {
            self.sceneView.scene.rootNode.addChildNode(self.focusSquare)
            self.focusSquare.state = .detecting(hitTestResult: result, camera: camera)
        }
    } else {
        DispatchQueue.main.async {
            self.focusSquare.state = .initializing
            self.sceneView.pointOfView?.addChildNode(self.focusSquare)
        }
    }
}

And add some control Functions:

func hideFocusSquare()  { DispatchQueue.main.async { self.updateFocusSquare(isObjectVisible: true) } }  // to hide the focus square
func showFocusSquare()  { DispatchQueue.main.async { self.updateFocusSquare(isObjectVisible: false) } } // to show the focus square

From the VirtualObjectARView.swift COPY! the entire function smartHitTest to the ViewController.swift (so they exist twice)

func smartHitTest(_ point: CGPoint,
                  infinitePlane: Bool = false,
                  objectPosition: float3? = nil,
                  allowedAlignments: [ARPlaneAnchor.Alignment] = [.horizontal, .vertical]) -> ARHitTestResult? {

    // Perform the hit test.
    let results = sceneView.hitTest(point, types: [.existingPlaneUsingGeometry, .estimatedVerticalPlane, .estimatedHorizontalPlane])

    // 1. Check for a result on an existing plane using geometry.
    if let existingPlaneUsingGeometryResult = results.first(where: { $0.type == .existingPlaneUsingGeometry }),
        let planeAnchor = existingPlaneUsingGeometryResult.anchor as? ARPlaneAnchor, allowedAlignments.contains(planeAnchor.alignment) {
        return existingPlaneUsingGeometryResult
    }

    if infinitePlane {

        // 2. Check for a result on an existing plane, assuming its dimensions are infinite.
        //    Loop through all hits against infinite existing planes and either return the
        //    nearest one (vertical planes) or return the nearest one which is within 5 cm
        //    of the object's position.
        let infinitePlaneResults = sceneView.hitTest(point, types: .existingPlane)

        for infinitePlaneResult in infinitePlaneResults {
            if let planeAnchor = infinitePlaneResult.anchor as? ARPlaneAnchor, allowedAlignments.contains(planeAnchor.alignment) {
                if planeAnchor.alignment == .vertical {
                    // Return the first vertical plane hit test result.
                    return infinitePlaneResult
                } else {
                    // For horizontal planes we only want to return a hit test result
                    // if it is close to the current object's position.
                    if let objectY = objectPosition?.y {
                        let planeY = infinitePlaneResult.worldTransform.translation.y
                        if objectY > planeY - 0.05 && objectY < planeY + 0.05 {
                            return infinitePlaneResult
                        }
                    } else {
                        return infinitePlaneResult
                    }
                }
            }
        }
    }

    // 3. As a final fallback, check for a result on estimated planes.
    let vResult = results.first(where: { $0.type == .estimatedVerticalPlane })
    let hResult = results.first(where: { $0.type == .estimatedHorizontalPlane })
    switch (allowedAlignments.contains(.horizontal), allowedAlignments.contains(.vertical)) {
    case (true, false):
        return hResult
    case (false, true):
        // Allow fallback to horizontal because we assume that objects meant for vertical placement
        // (like a picture) can always be placed on a horizontal surface, too.
        return vResult ?? hResult
    case (true, true):
        if hResult != nil && vResult != nil {
            return hResult!.distance < vResult!.distance ? hResult! : vResult!
        } else {
            return hResult ?? vResult
        }
    default:
        return nil
    }
}

You might see some errors in the copied function regarding the hitTest. Just correct it like so:

hitTest... // which gives an Error
sceneView.hitTest... // this should correct it

Implement the renderer updateAtTime function and add this lines:

func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
    // For the Focus Square
    if isFocusSquareEnabled { showFocusSquare() }

    self.updateObjectToCurrentTrackingPosition() // *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
}

And finally add some helper functions for the Focus Square

func hideFocusSquare() { DispatchQueue.main.async { self.updateFocusSquare(isObjectVisible: true) } }  // to hide the focus square
func showFocusSquare() { DispatchQueue.main.async { self.updateFocusSquare(isObjectVisible: false) } } // to show the focus square

At this point you might still see about a dozen errors and warnings in the imported files, this might occur, when doing this in Swift 5 and you have some Swift 4 files. Just let Xcode correct the errors. (Its all about renaming some code statements, Xcode knows best)

Go in VirtualObject.swift and search for this code block:

if smoothMovement {
    let hitTestResultDistance = simd_length(positionOffsetFromCamera)

    // Add the latest position and keep up to 10 recent distances to smooth with.
    recentVirtualObjectDistances.append(hitTestResultDistance)
    recentVirtualObjectDistances = Array(recentVirtualObjectDistances.suffix(10))

    let averageDistance = recentVirtualObjectDistances.average!
    let averagedDistancePosition = simd_normalize(positionOffsetFromCamera) * averageDistance
    simdPosition = cameraWorldPosition + averagedDistancePosition
} else {
    simdPosition = cameraWorldPosition + positionOffsetFromCamera
}

Outcomment or replace this entire block by this single line of code:

simdPosition = cameraWorldPosition + positionOffsetFromCamera

At this point you should be able to compile the project and run it on a device. You should see the Spaceship and a yellow focus square that should already work.

To start placing an Object, that you can drag you need some function to create a so called VirtualObject as I said in the beginning.

Use this example function to test (add it somewhere in the view controller):

override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {

    if focusSquare.state != .initializing {
        let position = SCNVector3(focusSquare.lastPosition!)

        // *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
        let testObject = VirtualObject() // give it some name, when you dont have anything to load
        testObject.geometry = SCNCone(topRadius: 0.0, bottomRadius: 0.2, height: 0.5)
        testObject.geometry?.firstMaterial?.diffuse.contents = UIColor.red
        testObject.categoryBitMask = 0b00000010
        testObject.name = "test"
        testObject.castsShadow = true
        testObject.position = position

        sceneView.scene.rootNode.addChildNode(testObject)
    }
}

Note: everything you want to drag on a plane, must be setup using VirtualObject() instead of SCNNode(). Everything else regarding the VirtualObject stays the same as SCNNode

(You can also add some common SCNNode extensions as well, like the one to load scenes by its name - useful when referencing imported models)

Have fun!

Answer 2

I added some of my ideas to Claessons's answer. I noticed some lag when dragging the node around. I found that the node cannot follow the finger's movement.

To make the node move more smoothly, I added a variable that keeps track of the node that is currently being moved, and set the position to the location of the touch.

    var selectedNode: SCNNode?

Also, I set a .categoryBitMask value to specify the category of nodes that I want to edit(move). The default bit mask value is 1.

The reason why we set the category bit mask is to distinguish between different kinds of nodes, and specify those that you wish to select (to move around, etc).

    enum CategoryBitMask: Int {
        case categoryToSelect = 2        // 010
        case otherCategoryToSelect = 4   // 100
        // you can add more bit masks below . . .
    }

Then, I added a UILongPressGestureRecognizer in viewDidLoad().

        let longPressRecognizer = UILongPressGestureRecognizer(target: self, action: #selector(longPressed))
        self.sceneView.addGestureRecognizer(longPressRecognizer)

The following is the UILongPressGestureRecognizer I used to detect a long press, which initiates the dragging of the node.

First, obtain the touch location from the recognizerView

    @objc func longPressed(recognizer: UILongPressGestureRecognizer) {

       guard let recognizerView = recognizer.view as? ARSCNView else { return }
       let touch = recognizer.location(in: recognizerView)

The following code runs once when a long press is detected.

Here, we perform a hitTest to select the node that has been touched. Note that here, we specify a .categoryBitMask option to select only nodes of the following category: CategoryBitMask.categoryToSelect

       // Runs once when long press is detected.
       if recognizer.state == .began {
            // perform a hitTest
            let hitTestResult = self.sceneView.hitTest(touch, options: [SCNHitTestOption.categoryBitMask: CategoryBitMask.categoryToSelect])

            guard let hitNode = hitTestResult.first?.node else { return }

            // Set hitNode as selected
            self.selectedNode = hitNode

The following code will run periodically until the user releases the finger. Here we perform another hitTest to obtain the plane you want the node to move along.

        // Runs periodically after .began
        } else if recognizer.state == .changed {
            // make sure a node has been selected from .began
            guard let hitNode = self.selectedNode else { return }

            // perform a hitTest to obtain the plane 
            let hitTestPlane = self.sceneView.hitTest(touch, types: .existingPlane)
            guard let hitPlane = hitTestPlane.first else { return }
            hitNode.position = SCNVector3(hitPlane.worldTransform.columns.3.x,
                                           hitNode.position.y,
                                           hitPlane.worldTransform.columns.3.z)

Make sure you deselect the node when the finger is removed from the screen.

        // Runs when finger is removed from screen. Only once.
        } else if recognizer.state == .ended || recognizer.state == .cancelled || recognizer.state == .failed{

            guard let hitNode = self.selectedNode else { return }

            // Undo selection
            self.selectedNode = nil
        }
    }