SwiftUI: Detect finger position on Mac trackpad

Question

I'm making a SwiftUI app for macOS and I'd like to use the trackpad as an (x, y) input by detecting the position of the user's fingers. I wanna be able to detect multiple fingers that are resting on the trackpad (not dragging). How do I do that?

A similar question has been asked before, but I'm asking again because that was from nearly 10 years ago, the answers are all in Obj-C (one in Swift 3), and I'm wondering if there's an updated methodology. Most importantly, I've no clue how to implement the Obj-C code into my SwiftUI app so if there isn't any updated methodology, I'd appreciate if someone could just explain how to implement the old Obj-C code.

To demonstrate what I mean, this video demo of the AudioSwift app does exactly what I want. macOS itself also uses this for hand-writing Chinese (although I don't need to recognize characters).

Answer 1

Always split your task into smaller ones and do them one by one. Ask in the same way and avoid broad questions touching lot of topics.

Goal

• Track pad view (gray rectangle)
• Circles on top of it showing fingers physical position

Step 1 - AppKit

• SwiftUI doesn't provide all the required information
• AppKit & NSTouch does - normalizedPosition.

First step is to create a simple AppKitTouchesView forwarding required touches via a delegate.

import SwiftUI
import AppKit

protocol AppKitTouchesViewDelegate: AnyObject {
    // Provides `.touching` touches only.
    func touchesView(_ view: AppKitTouchesView, didUpdateTouchingTouches touches: Set<NSTouch>)
}

final class AppKitTouchesView: NSView {
    weak var delegate: AppKitTouchesViewDelegate?

    override init(frame frameRect: NSRect) {
        super.init(frame: frameRect)
        // We're interested in `.indirect` touches only.
        allowedTouchTypes = [.indirect]
        // We'd like to receive resting touches as well.
        wantsRestingTouches = true
    }

    required init?(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    private func handleTouches(with event: NSEvent) {
        // Get all `.touching` touches only (includes `.began`, `.moved` & `.stationary`).
        let touches = event.touches(matching: .touching, in: self)
        // Forward them via delegate.
        delegate?.touchesView(self, didUpdateTouchingTouches: touches)
    }

    override func touchesBegan(with event: NSEvent) {
        handleTouches(with: event)
    }

    override func touchesEnded(with event: NSEvent) {
        handleTouches(with: event)
    }

    override func touchesMoved(with event: NSEvent) {
        handleTouches(with: event)
    }

    override func touchesCancelled(with event: NSEvent) {
        handleTouches(with: event)
    }
}

Step 2 - Simplified touch structure

Second step is to create a simple custom Touch structure which holds all the required information only and is SwiftUI compatible (not flipped y).

struct Touch: Identifiable {
    // `Identifiable` -> `id` is required for `ForEach` (see below).
    let id: Int
    // Normalized touch X position on a device (0.0 - 1.0).
    let normalizedX: CGFloat
    // Normalized touch Y position on a device (0.0 - 1.0).
    let normalizedY: CGFloat

    init(_ nsTouch: NSTouch) {
        self.normalizedX = nsTouch.normalizedPosition.x
        // `NSTouch.normalizedPosition.y` is flipped -> 0.0 means bottom. But the
        // `Touch` structure is meants to be used with the SwiftUI -> flip it.
        self.normalizedY = 1.0 - nsTouch.normalizedPosition.y
        self.id = nsTouch.hash
    }
}

Step 3 - Wrap it for the SwiftUI

• NSViewRepresentable documentation
• Binding documentation

Third step is to create a SwiftUI view wrapping our AppKit AppKitTouchesView view.

struct TouchesView: NSViewRepresentable {
    // Up to date list of touching touches.
    @Binding var touches: [Touch]

    func updateNSView(_ nsView: AppKitTouchesView, context: Context) {
    }

    func makeNSView(context: Context) -> AppKitTouchesView {
        let view = AppKitTouchesView()
        view.delegate = context.coordinator
        return view
    }

    func makeCoordinator() -> Coordinator {
        Coordinator(self)
    }

    class Coordinator: NSObject, AppKitTouchesViewDelegate {
        let parent: TouchesView

        init(_ view: TouchesView) {
            self.parent = view
        }

        func touchesView(_ view: AppKitTouchesView, didUpdateTouchingTouches touches: Set<NSTouch>) {
            parent.touches = touches.map(Touch.init)
        }
    }
}

Step 4 - Make a TrackPadView

Fourth step is to create a TrackPadView which internally does use our TouchesView and draws circles on it representing physical location of fingers.

struct TrackPadView: View {
    private let touchViewSize: CGFloat = 20

    @State var touches: [Touch] = []

    var body: some View {
        ZStack {
            GeometryReader { proxy in
                TouchesView(touches: self.$touches)

                ForEach(self.touches) { touch in
                    Circle()
                        .foregroundColor(Color.green)
                        .frame(width: self.touchViewSize, height: self.touchViewSize)
                        .offset(
                            x: proxy.size.width * touch.normalizedX - self.touchViewSize / 2.0,
                            y: proxy.size.height * touch.normalizedY - self.touchViewSize / 2.0
                        )
                }
            }
        }
    }
}

Step 5 - Use it in the main ContentView

Fifth step is to use it in our main view with some aspect ratio which is close to the real trackpad aspect ratio.

struct ContentView: View {
    var body: some View {
        TrackPadView()
            .background(Color.gray)
            .aspectRatio(1.6, contentMode: .fit)
            .padding()
            .frame(maxWidth: .infinity, maxHeight: .infinity)
    }
}

Complete project

• Open Xcode
• Create a new project (macOS App & Swift & SwiftUI)
• Copy & paste ContentView.swift from this gist