Add 2D or 3D Face Filters like MSQRD/SnapChat Using Google Vision API for iOS

Question

Here's some research I have done so far: - I have used Google Vision API to detect various face landmarks. Here's the reference: https://developers.google.com/vision/introduction

• Here's the link to Sample Code to get the facial landmarks. It uses the same Google Vision API. Here's the reference link: https://github.com/googlesamples/ios-vision

• I have gone through the various blogs on internet which says MSQRD based on the Google's cloud vision. Here's the link to it: https://medium.com/@AlexioCassani/how-to-create-a-msqrd-like-app-with-google-cloud-vision-802b578b30a0

• For Android here's the reference: https://www.raywenderlich.com/158580/augmented-reality-android-googles-face-api

• There are multiple paid SDK's which full fills the purpose. But they are highly priced. So cant able to afford it. For instance:

1) https://deepar.ai/contact/

2) https://www.luxand.com/

There is possibility might have some see this question as duplicate of this: Face filter implementation like MSQRD/SnapChat

But the thread is almost 1.6 years old with no right answers to it.

I have gone through this article: https://dzone.com/articles/mimic-snapchat-filters-programmatically-1

It describes all the essential steps to achieve the desired results. But they advice to use their own made SDK.

As per my research no good enough material is around which helps to full fill the desired results like MSQRD face filters.

One more Github repository around which has same implementation but it doesn't gives much information about same. https://github.com/rootkit/LiveFaceMask

Now my question is:

If we have the facial landmarks using Google Vision API (or even using DiLib), how I can add 2d or 3d models over it. In which format this needs to be done like this require some X,Y coordinates with vertices calculation.

NOTE: I have gone through the Googles "GooglyEyesDemo" which adds the preview layer over eyes. It basically adds a view over the face. So I dont want to add UIView one dimensional preview layers over it. Image attached for reference :

https://developers.google.com/vision/ios/face-tracker-tutorial

Creating Models: I also want to know how to create models for live filters like MSQRD. I welcome any software or format recommendations. Hope the research I have done will help others and someone else experience helps me to achieve the desired results. Let me know if any more details are required.**

Image attached for more reference:

Thanks Harry

Answer 1

Canvas class is used in android for drawing such 3D / 2D models or core graphics for IOS can be used. What you can do is detect the face components, take their location points and draw images on top of them. Consider going through this