How to understand head pose estimation angles in Python with OpenCV?

Question

I am working through a Python and OpenCV head pose estimation tutorial found here:

https://www.learnopencv.com/head-pose-estimation-using-opencv-and-dlib/

I am able to accurately project a 3D point onto the 2D image. However, I am unable to understand the practical meaning of the Euler angles (yaw, pitch, roll) that I calculate using cv2.decomposeProjectionMatrix

I need to know whether or not the values correspond to (yaw, pitch, roll) or (roll, pitch, yaw), etc. Also, I need to understand the axis orientation used so that I can know where the degrees of rotation is measured from.

Output Image: https://www.learnopencv.com/wp-content/uploads/2016/09/head-pose-example-1024x576.jpg

Output Angles: [[-179.30011146], [ 53.77756583], [-176.6277211 ]]

Here is my code

# --- Imports ---
import cv2
import numpy as np
# --- Main ---
if __name__ == "__main__":

    # Read Image
    im = cv2.imread("headPose.jpg");
    size = im.shape

    #2D image points. If you change the image, you need to change vector
    image_points = np.array([
                                (359, 391),     # Nose tip
                                (399, 561),     # Chin
                                (337, 297),     # Left eye left corner
                                (513, 301),     # Right eye right corne
                                (345, 465),     # Left Mouth corner
                                (453, 469)      # Right mouth corner
                            ], dtype="double")

    # 3D model points.
    model_points = np.array([
                                (0.0, 0.0, 0.0),             # Nose tip
                                (0.0, -330.0, -65.0),        # Chin
                                (-225.0, 170.0, -135.0),     # Left eye left corner
                                (225.0, 170.0, -135.0),      # Right eye right corne
                                (-150.0, -150.0, -125.0),    # Left Mouth corner
                                (150.0, -150.0, -125.0)      # Right mouth corner
                            ])

    # Camera internals
    focal_length = size[1]
    center = (size[1]/2, size[0]/2)
    camera_matrix = np.array(
                             [[focal_length, 0, center[0]],
                             [0, focal_length, center[1]],
                             [0, 0, 1]], dtype = "double"
                             )

    # Lens distortion - assumed to be zero
    dist_coeffs = np.zeros((4,1)) 

    # Calculate perspective and point
    (_, rvec, tvec) = cv2.solvePnP(model_points, image_points, camera_matrix, dist_coeffs, flags=cv2.SOLVEPNP_ITERATIVE)

    # Calculate Euler angles
    rmat = cv2.Rodrigues(rvec)[0] # rotation matrix
    pmat = np.hstack((rmat, tvec)) # projection matrix
    eulers = cv2.decomposeProjectionMatrix(pmat)[-1]
    print(eulers)

    # Projecting a 3D point
    ## features
    for p in image_points:
        cv2.circle(im, (int(p[0]), int(p[1])), 3, (0,0,255), -1)

    ## projection of multiple points
    proj = np.array([(0., 0., 1000.)])
    (poi1, jacobian1) = cv2.projectPoints(model_points[0]+proj, rvec, tvec, camera_matrix, dist_coeffs)

    ## 2D space    
    p1 = ( int(image_points[0][0]), int(image_points[0][1]))
    c1 =  ( int(poi1[0][0][0]), int(poi1[0][0][1]))

    cv2.line(im, p1, c1, (255,0,0), 2)

    # Display image
    cv2.imshow("Output", im)
    cv2.waitKey(0)

Test Image: https://www.learnopencv.com/wp-content/uploads/2016/09/headPose.jpg

Thanks!

Answer 1

Here's method to convert rotation matrix to euler angles (roll, pitch, yaw). Orientation depends on model_points and seems to face in the camera's direction upside-down (so the person looking at the camera should have yaw and roll ~180 degrees)

Code from the article (with x,y,z renamed to roll, pitch, yaw). Angles are in radians.

# Calculates rotation matrix to euler angles
# The result is the same as MATLAB except the order
# of the euler angles ( roll and yaw are swapped ).
def rotationMatrixToEulerAngles(R) :
    
    sy = math.sqrt(R[0,0] * R[0,0] +  R[1,0] * R[1,0])
    
    singular = sy < 1e-6

    if  not singular :
        roll = math.atan2(R[2,1] , R[2,2])
        pitch = math.atan2(-R[2,0], sy)
        yaw = math.atan2(R[1,0], R[0,0])
    else :
        roll = math.atan2(-R[1,2], R[1,1])
        pitch = math.atan2(-R[2,0], sy)
        yaw = 0

    return np.array([roll, pitch, yaw])