Can CATransform3D be used to get eye size dimensions in Face Mesh?

Question

I am trying to get width of the eyes and distance of 2 eyes using 3D Face Mesh of ARKit.

I have used CATransform3D of ARAnchor;

 struct CATransform3D
{
  CGFloat m11, m12, m13, m14;
  CGFloat m21, m22, m23, m24;
  CGFloat m31, m32, m33, m34;
  CGFloat m41, m42, m43, m44;
};

Below is my code;

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {

guard let faceAnchor = anchor as? ARFaceAnchor else { return }

let leftcaTransform3DValue : CATransform3D = (faceAnchor.blendShapes[.eyeBlinkLeft]?.caTransform3DValue)!

let rightcaTransform3DValue : CATransform3D = (faceAnchor.blendShapes[.eyeBlinkRight]?.caTransform3DValue)!

print("  m11 : \(String(describing:leftcaTransform3DValue.m11)) m12 : \(String(describing:leftcaTransform3DValue.m12)) m13 : \(String(describing:leftcaTransform3DValue.m13)) m14 : \(String(describing:leftcaTransform3DValue.m14)) m21 : \(String(describing:leftcaTransform3DValue.m21)) m22 : \(String(describing:leftcaTransform3DValue.m22)) m23 : \(String(describing:leftcaTransform3DValue.m23)) m24 : \(String(describing:leftcaTransform3DValue.m24)) m31 : \(String(describing:leftcaTransform3DValue.m31)) m32 : \(String(describing:leftcaTransform3DValue.m32)) m33 : \(String(describing:leftcaTransform3DValue.m33)) m34 : \(String(describing:leftcaTransform3DValue.m34)) m41 : \(String(describing:leftcaTransform3DValue.m41)) m42 : \(String(describing:leftcaTransform3DValue.m42)) m43 : \(String(describing:leftcaTransform3DValue.m43)) m44 : \(String(describing:leftcaTransform3DValue.m44)) " )
}

And as a result of leftcaTransform3DValue I got values like;

m11 =  -5.22553711590422e-315
...
...
...
m44 =   2.13285635582599e-314

Same for the rightcaTransform3DValue.

So my question is do these values specify any dimension or size measurement?

Can I calculate width of the eyes and distance between two eyes?

Any help is really appreciated.

Answer 1

So my question is do these values specify any dimension or size measurement?

No. The numbers you’re getting are nonsense, because the way you’re getting them is... maybe not quite nonsense, but pretty close.

The blendShapes dictionary on ARFaceAnchor is documented as having values of type NSNumber, where the underlying numeric value of the NSNumber is a Float between 0.0 and 1.0.

NSNumber is an object wrapper for many possible types of scalar numeric values. It has methods for fetching its underlying value as various types (by converting to a different representation of the same number). But given that these particular numbers are documented to be floating-point values between 0 and 1, there’s not much sense to fetching the intValue or boolValue, etc.

NSNumber is a subclass of NSValue, which is an object wrapper for many kinds of types not otherwise expressible as objects — ranges, sizes, pointers, and 3D transform matrices, among others. Those types can’t be converted between each other like numbers can, so the only type that’s meaningful to fetch out of an NSValue is the type it was created with. Any other type gives you nonsense.

Back to blendShapes — it’s further documented that each blend shape value in the dictionary isn’t just a number, but a number that tells you the progress of an animation parameter. eyeBlinkLeft doesn’t claim to tell you anything about where or how big the left eye is — it tells you how “blinked” (closed) the left eyelid is.

You’re barking up the wrong tree, but if you look at the docs for the classes and properties you’re using, you’ll be better able to make educated guesses later.

Can I calculate width of the eyes and distance between two eyes?

Update: In "ARKit 2", aka ARKit in iOS 12, the leftEyeTransform and rightEyeTransform provide the 3D position (relative to the face anchor) of the center of each eyeball. (Also the orientation of each eye.) That might help your use case, but if what you're actually after has to do with the position/size of the pupils or the eye openings...

There’s no API that’ll do this for you. ARKit does provide information you could use to find it out yourself, but not in a way that’s guaranteed to always work.

ARFaceGeometry gives you a triangle mesh that maps a few hundred or so points on the face, in a way that’s topologically stable within a session. That is, for example, assuming the 57th vertex in the mesh is the tip of the nose, it’ll stay on the tip of the nose even as the face wrinkles and stretches and that point’s position relative to other points changes.

The problems:

• The API doesn’t tell you which vertices (points in the mesh) are which (in terms of face landmarks like eye corner, nose tip, etc).
• The topology of the mesh is stable within a session, but Apple doesn’t guarantee it won’t change between iOS versions, devices, etc.

So, though some experimentation you might be able to work out which vertices are the inside corner of the left eye, the outside corner of the left eye, etc. Once you do that, you can look at their positions to estimate useful quantities like eye width, inter-pupillary distance, etc. However, those measurements are based on assumptions about the mesh that might not always hold, so you don’t know when it’ll break for users of your app.