How to detect the line of sight of a person using kinect?

Question

Currently I am working on a project using Kinect which requires me to know the where the person is looking at that time, for which I figured out I need to find the line of sight of that person.

Right now, I can find the head point of the skeleton of the person but can't track the eye movement.

if (body.TrackingState == SkeletonTrackingState.Tracked)
{
    Joint joint = body.Joints[JointType.Head];
    SkeletonPoint skeletonPoint = joint.Position;

    // 2D coordinates in pixels
    System.Drawing.Point point = new System.Drawing.Point();

    if (_mode == CameraMode.Color)
    {
        // Skeleton-to-Color mapping
        ColorImagePoint colorPoint = _sensor.CoordinateMapper.MapSkeletonPointToColorPoint(skeletonPoint, ColorImageFormat.RgbResolution640x480Fps30);

        point.X = colorPoint.X;
        point.Y = colorPoint.Y;
        //Console.WriteLine(" X == " + point.X + " Y == " + point.Y);
        X = (int)Math.Floor(point.X + 0.5);
        Y = (int)Math.Floor(point.Y + 0.5);
    }

    // DRAWING...
    Ellipse ellipse = new Ellipse
    {
        Fill = System.Windows.Media.Brushes.LightBlue,
        Width = 20,
        Height = 20
    };

     Canvas.SetLeft(ellipse, point.X - ellipse.Width / 2);
     Canvas.SetTop(ellipse, point.Y - ellipse.Height / 2);

     canvas.Children.Add(ellipse);
 }

Here point.X and point.Y are the head points of the skeleton.

Answer 1

Have you looked at the FaceBasics Sample Project?

I believe you want to use the FaceFrameSource / FaceFrameReader (note: not HDFace). You'll be able to get the face orientation as a Quarternion (and the sample project translates that to Pitch/Yaw/Roll).

Combining that with the 3D location of the head from the skeleton, I think you should be able to create an approximate line of sight.

The How-to Videos cover Face including some information on Orientation (5th video, skip in about 18:20 - your specific question is asked at 21:49).

EDIT: Rough proof of concept showing modifications made to FaceBasics Sample Project - added to ~line 565, right after the face info is drawn (I also needed to change the scope of pitch/yaw/roll defined a few lines above and set their default values to 0). This creates a circle for a head, and a yellow line looking at an approximate gaze location.

Joint HeadJoint = this.bodies[faceIndex].Joints[JointType.Head];
ColorSpacePoint colorPoint = this.coordinateMapper.MapCameraPointToColorSpace(HeadJoint.Position);
Point HeadPoint = new Point(colorPoint.X, colorPoint.Y);
Point GazePoint = new Point(HeadPoint.X - Math.Sin((double)yaw * 0.0175) * 600, HeadPoint.Y - Math.Sin((double)pitch * 0.0175) * 600);
drawingContext.DrawLine(new Pen(System.Windows.Media.Brushes.Yellow, 5), HeadPoint, GazePoint);
drawingContext.DrawEllipse(System.Windows.Media.Brushes.LightBlue, null, HeadPoint, 70, 70);

EDIT 2: Just saw your new comment saying that you are using SDK v1.8 - my answer is going off of v2.0, and I can't speak for how things would be different with the older SDK/Sensor.