Translate/Rotate 2D points to change perspective

Question

I am recording a video of a users eye, and then using computer vision to track their eyes in an attempt to estimate their gaze, however the angle I am able to record the user at is not straight on and the representation of the data needs to be displayed, as if the user is looking straight on.

To explain myself further please consider the following images depicticing what I have so far, and what I am trying to achieve:

I thought perhaps the best way to achieve this would be to translate the perspective, but not being very well versed in this, I have no idea where to begin.

I am open to any suggestions on the best way to achieve the desired result, but please bear in mind that my matrix math is fairly rusty, so if you use any well-known methods, please cater to my ignorance and explain everything as well as you can.

The data is currently stored as a NumPy array of X/Y Points

Answer 1

My matrix math is worse than "fairly rusty," but I've got a few ideas that could be helpful.

Overall, there's a lot more information on transforming images than transforming discrete points. You may want to look into transforming the eye portion of your image rather than transforming the pupil points.

Anyway, here are my ideas:

Approach 1: simple perspective transformation

This answer describes how to perform perspective transformation in Python Imaging Library, using numpy to calculate coefficients for the transformation.

It's probably easy to adapt this to operate on points rather than an image (google suggests scipy has some functions similar to PIL.Image.transform` which may be more applicable). Failing this, you could just render your points onto a binary image, as white pixels on a black background, then transforming that image and reading the points back out.

However, for perspective transformation, you still need an approach for determining the coordinates of pa, the plane from which you're transforming. You could likely achieve reasonable results simply by fitting a rectangle around the eye. To do this, I would probably rotate your figure so it was parallel with the X axis, by constructing a line between the ends of the eye and then rotating by the line's angle from 0°. Then I'd record the bounding box, and rotate both back. Your plane will look something like this:

At this point, you might be able to extract the angle of the dominant contours of the eye and pinch and squeeze your bounding rectangle accordingly. Simple perspective transformation will probably prove unreliable, though.

Approach 2: Better perspective transformation

Given a set of starting points and a set of ending points, there's almost certainly a way to calculate perspective transformation coefficients from those, even if the number is greater than 4. You could just skip bounding boxes and assume translating each point to its counterpart on the ideal shape, then calculate coefficients based on that. Don't ask me how, though, I have no idea :P

Approach 3: basic stretching

If your "destination shape" has the same number of points as the shape from which you're transforming, you could emulate the stretch functions of many image editing programs. Photoshop has tools that allow you to pull points on a shape to move them around, stretching the content inside. If you could reproduce this behavior, you could just move each point on the starting shape to a corresponding point on the destination shape, stretching the image. This is probably the most reliable approach, simply stretch your image to fit the destination shape, then pull the pupil from that new image.

---

The problem with all these approaches is that normal perspective transformation will never be quite accurate, because the eye is curved, not flat. You can't really approximate the surface of an eye with a plane and expect complete accuracy. Even stretching (approach 3) will suffer from the angle of your photo; it will favor the visible side of the eye and make it appear as if the eye is looking far more to the left (their right) than it is. If the angle of the photo is constant and known, you may be able to correct this yourself. Otherwise, I don't see an easy solution to this obstacle.

---

I know very little about higher-level math, but hopefully you find my ideas helpful.

FWIW, eye tracking is well-researched and there are several thorough papers like this one