Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

'Project' ARFaceAnchor points to screen

I'm trying to superimpose ARFaceAnchor vertices on-screen to accomplish two scenarios 1) have a virtual face maintain is center (onscreen) position, but reflect changes in geometry.vertices 2) have the virtual face overlap the actual face (from previewlayer).

I've followed Rickster's advice here, but have only succeeded in projecting the face from certain angles onscreen (only appears lower left and rotates) . I'm not too familiar with the different purposes of each matrix, but this is where I've gotten so far. Any advice?

let modelMatrix = faceAnchor.transform
var points: [CGPoint] = []

faceAnchor.geometry.vertices.forEach {
    // Convert the vertex position from model space to camera space (use the anchor’s transform)
    let vertex4 = vector_float4($0.x, $0.y, $0.z, 1)
    let vertexCamera = simd_mul(modelMatrix, vertex4)

    // Multiply with the camera projection with that vector to get to normalized image coordinates 
    let normalizedImageCoordinates = simd_mul(projectionMatrix, vertexCamera)

    let point = CGPoint(x: CGFloat(normalizedImageCoordinates.x), y: CGFloat(normalizedImageCoordinates.y))
    points.append(point)
} 
like image 791
jjc Avatar asked Feb 13 '18 01:02

jjc


1 Answers

For those whom are interested, here's the solution to (2) - you can normalize the points to keep the face centered for (1)

let faceAnchors = anchors.flatMap { $0 as? ARFaceAnchor }

guard !faceAnchors.isEmpty,
    let camera = session.currentFrame?.camera,
    let targetView = SomeUIView() else { return }

// Calculate face points to project to screen

let projectionMatrix = camera.projectionMatrix(for: .portrait, viewportSize: targetView.bounds.size, zNear: 0.001, zFar: 1000)  // A transform matrix appropriate for rendering 3D content to match the image captured by the camera
let viewMatrix = camera.viewMatrix(for: .portrait)        // Returns a transform matrix for converting from world space to camera space.

let projectionViewMatrix = simd_mul(projectionMatrix, viewMatrix)

for faceAnchor in faceAnchors  {

    let modelMatrix = faceAnchor.transform                  //  Describes the face’s current position and orientation in world coordinates; that is, in a coordinate space relative to that specified by the worldAlignment property of the session configuration. Use this transform matrix to position virtual content you want to “attach” to the face in your AR scene.
    let mvpMatrix = simd_mul(projectionViewMatrix, modelMatrix)

    // Calculate points

    let points: [CGPoint] = faceAnchor.geometry.vertices.flatMap({ (vertex) -> CGPoint? in

        let vertex4 = vector_float4(vertex.x, vertex.y, vertex.z, 1)

        let normalizedImageCoordinates = simd_mul(mvpMatrix, vertex4)

        return CGPoint(x: CGFloat(normalizedImageCoordinates.x ),
                       y: CGFloat(normalizedImageCoordinates.y ))
    })

}
like image 149
jjc Avatar answered Nov 03 '22 04:11

jjc