Facial Features Detection and Tracking

Question

I want to detect and track lips and eye corners in a 2D video

I have completed the facial features detection in frontal pose, neutral expressions and good light conditions using OpenCV Viola-Jones Face and Feature points detector.

I used CascadeClassifier::detectMultiScale with the haarcascade_frontalface_alt, haarcascade_eye, haarcascade_mcs_mouth xml files. It works fine upto 15 degrees of rotation and neutral expression.

But now, I just want to know if detection of facial features (eyes, mouth as in the image) is possible in below conditions ??

• Head rotation upto 45 degrees

• Expressions - Talking, laughing, smiling, ...

  

OpenCV is not a must, any library or implementation is fine. For a start, I am ok with any reference paper also. 3D Range camera and other 3D camera sensors cannot be used.

Answer 1

You've already posted a question here: Facial Feature Points Detection using OpenCV . And I'm tempted to call this a duplicate of your own question.

This already answers your question using the popular flandmark tool. The only difference in this question is that you've asked about larger degrees of pose variation, i.e. up to 45 degrees. If you read the associated paper with flandmark, they evaluate on one of the current state-of-the-art datasets for evaluation, Labeled Faces in the Wild (LFW) which covers a moderate range of pose variation already.

If you're willing to sacrifice speed for accuracy on off-pose faces, then another work in 2012 has far denser landmarking algorithm evaluated over an images set that I would argue has more pose variation than LFW. The work's project webpage is here.

I also recommend in the future that you reference your own questions if they are this similar to each other and clearly delineate what is different and unique about your question that merits a new one.