rotate face from non-front to front opencv

Question

I've struggled for a week with a problem that I can't rotate a face from non-front like the Left Face in this image to front like the Right Face in the same image.

1). I use a tutorial about delaunay at learnopencv website to get all the triangle of the Left Face (about 138 triangles) like this:

...

triangle num 4: [100, 383] | [120, 374] | [112, 385]

triangle num 5: [120, 374] | [100, 383] | [92, 345]

triangle num 6: [136, 396] | [137, 379] | [151, 394]

triangle num 7: [137, 379] | [136, 396] | [118, 395]

triangle num 8: [92, 345] | [86, 414] | [80, 451]

triangle num 9: [86, 414] | [92, 345] | [100, 383]

triangle num 10: [86, 414] | [100, 383] | [118, 395]

triangle num 11: [116, 336] | [120, 374] | [92, 345]

triangle num 12: [120, 374] | [116, 336] | [133, 344]

...

2). and this library: https://github.com/chili-epfl/attention-tracker (which can estimate head pose) to get a 4x4 matrix transformation of head pose of the Left Face, this matrix seems to be a result of the function Rodrigues in opencv.

 0.659585  -0.749291 -0.0592507  0.0591621 

-0.0327269 -0.107385 0.993679    -0.00523744 

-0.750917  -0.653477 -0.0953513  -0.280204 

 0         0         0           1 

So I don't know what is the next move to rotate all the triangle from step 1 with the 4x4 matrix from step 2 to get the result of the Right Face. Pls help!

Answer 1

How to project a face from non-frontal view to frontal view is a complex problem, even you have projected landmark position of the face. I suggest use two open source library that i use before.

The first is provided by MIT. this code isn't only face frontalizer and can use for any object. https://bitbucket.org/gbhuang/congealreal

The second library link is below that provided only for face projection.

https://github.com/eranid/adience_align

I think the second library is the most useful.