iOS11 ARKit: Can ARKit also capture the Texture of the user's face?

Question

I read the whole documentation on all ARKit classes up and down. I don't see any place that describes ability to actually get the user face's Texture.

ARFaceAnchor contains the ARFaceGeometry (topology and geometry comprised of vertices) and the BlendShapeLocation array (coordinates allowing manipulations of individual facial traits by manipulating geometric math on the user face's vertices).

But where can I get the actual Texture of the user's face. For example: the actual skin tone / color / texture, facial hair, other unique traits, such as scars or birth marks? Or is this not possible at all?

Answer 1

I've put together a demo iOS app that shows how to accomplish this. The demo captures a face texture map in realtime, applying it back to a ARSCNFaceGeometry to create a textured 3D model of the user's face.

Below you can see the realtime textured 3D face model in the top left, overlaid on top of the AR front facing camera view:

The demo works by rendering an ARSCNFaceGeometry, however instead of rendering it normally, you instead render it in texture space while continuing to use the original vertex positions to determine where to sample from in the captured pixel data.

Here are links to the relevant parts of the implementation:

• FaceTextureGenerator.swift — The main class for generating face textures. This sets up a Metal render pipeline to generate the texture.

• faceTexture.metal — The vertex and fragment shaders used to generate the face texture. These operate in texture space.

Almost all the work is done in a metal render pass, so it easily runs in realtime.

I've also put together some notes covering the limitations of the demo

---

If you instead want a 2D image of the user's face, you can try doing the following:

• Render the transformed ARSCNFaceGeometry to a 1-bit buffer to create an image mask. Basically you just want places where the face model appears to be white, while everything else should be black.

• Apply the mask to the captured frame image.

This should give you an image with just the face (although you will likely need to crop the result)