How to align face images c++ opencv

Question

I am developing a C++ application for face authentication. First, I have to detect the face and pre-process the image.

1. For face detection I have used the HaarCascadeClassifier. The problem is that the this tool or this algorithm gives me a facial region detected by a little bit large rectangle that englobes hair and some of the background. Is there a solution to change the dimension of this rectangle? I used "frontalfacecascaadclassifier.xml".
2. For face pre-processing i want to do face alignment exactly like this technique. How would I go about accomplishing this?

Answer 1

Finding the accurate position of the eyes in a given image is far from trivial. The Haar-cascades for finding the eyes in OpenCV produce too many false positive to be useful, moreover this approach won't be robust to image rotation (it may compensate slight rotation though, I don't know the training images). If I were you I'd start a breadth-first search on http://scholar.google.com for relevant papers of this research area.

You'll need a robust head pose estimation for aligning face images. I did some research myself and I think sharing algorithms and code is useful here. The most interesting approaches I have seen are:

• Gary B. Huang, Vidit Jain, and Erik Learned-Miller. Unsupervised joint alignment of complex images. International Conference on Computer Vision (ICCV), 2007. (Project page), (PDF Online available), (Source code)

• X. Zhu, D. Ramanan. Face Detection, Pose Estimation and Landmark Localization in the Wild Computer Vision and Pattern Recognition (CVPR) Providence, Rhode Island, June 2012. (Project page), (PDF Online available), (Source code)

Answer 2

Can't you then use another Haar classifier to find each eye (eyes are very easy to find) then assuming the person has two eyes and we define a 'level' face to mean the eyes are horizontal.

Simply measure the anlge between the two eyes and rotate the image by that angle.

angle = atan ( eye1.Y - eye2.Y ) / (eye1.X - eye2.X )

Answer 3

I tried the following face alignment code from the Labelled Faces in the Wild project page. It works really well and does not require detecting facial feature points. The C++ code can be downloaded from: http://vis-www.cs.umass.edu/faceAlignment/

If you still wish to find face key points, I find that the Viola-Jones detector is not very robust and accurate. I personally recommend using the Flandmark face keypoint detector: http://cmp.felk.cvut.cz/~uricamic/flandmark/ which is much much more robust and accurate. C code can be downloaded from the above site.

Answer 4

The state-of-the-art approach for face alignment must be this:

Supervised Descent Method and its Application to Face Alignment X. Xiong and F. De la Torre in CVPR 2013

It is extremely fast and effective. You can check their project website IntraFace.

They provide an easy-to-use software. However, the core part code, i.e supervised descent method (SDM) is not released, it is only simple linear regression which can be easily implemented.

A demo to show that it can handle tilted face is here (for privacy issue, add blur and pay attention to the axis in the top-left corner): https://drive.google.com/file/d/0BztytuqPViMoTG9QaEpZWi1NMGc/edit?usp=sharing