Getting the touched position in local coordinates of a transformed element

Question

I have an element which has been transformed with a matrix3d to give it perspective. It represents the screen of a handheld device.

There's a background image showing the handheld device itself, and this is not transformed. The element which holds this is positioned, and the screen element is positioned absolutely within it, at left: 0; top: 0;, and then transformed, with an origin in the top-left of the container. This was the easiest way for me to line it up perfectly with the background image (I used this very handy tool to come up with the matrix), and it moves the screen element away from the corner.

I want to be able to interact with the screen with mouse and touch events. To do this, on a click or touch event I need to find the coordinates in the local coordinate system of the screen element -- that is, the coordinates before the transform has taken place. In other words, when clicking the top-left of the handheld device's screen (which is not the top-left of its bounding box on the page!) I want [0, 0], and when clicking the top right of the screen, which in this transform's case is actually further up on the page as well as to the right, I want [untransformedWidth, 0].

Mouse events provide offsetX and offsetY which purportedly do exactly this (more on that below), but the touch events don't have these properties, so I need a way to calculate them myself.

Using math.js I can feed in the transformation matrix and invert it. I have some code to loop over the CSS rules to get the transform: matrix3d(...) rule (I don't want to repeat it in my code if I don't have to), which I'll skip over -- I know it works because the numbers match the CSS.

Note that CSS has its matrix elements in column order, so matrix3d(a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p) looks in regular matrix notation like this:

┌            ┐
│ a  e  i  m │
│ b  f  j  n │
│ c  g  k  o │
│ d  h  l  p │
└            ┘

Meanwhile, math.js wants its matrices declared row by row, like [[a, e, i, m], [b, f, j, n]....

So starting where I have a list of the number elements from inside the matrix3d(...) expression, in CSS order, I'm building and inverting the matrix like this:

var rows = [[], [], [], []];
for (var i = 0; i < elements.length; i++) {
  rows[i % 4][Math.floor(i / 4)] = elements[i];
}

var matrixTransform = math.matrix(rows);

var invertedMatrixTransform = math.inv(matrixTransform);

I then set up a mouse event handler on the screen element:

screen.addEventListener('mousedown', function (event) {
  var rect = container.getBoundingClientRect();
  var x = event.clientX - rect.left;
  var y = event.clientY - rect.top;

If I move a marker (relative to the container) to this point [x, y], it appears exactly where I clicked. So I know this much is working. I then multiply a vector of these coordinates by the inverse transformation matrix:

  var vector = math.matrix([x, y, 0, 1]);
  var result = math.multiply(inverseMatrixTransform, vector);

If I move another marker (this one relative to the screen element) to the resulting vector's values [result.get([0]), result.get([1])] it moves to roughly the same position as the previous marker, but it's not quite right. It seems that the further from the origin I go, the more error there is, until it's really quite bad towards the right and bottom edges.

But then what if I check against offsetX and offsetY? Well, it turns out that the answer depends on the browser.

In Firefox, the coordinates found with offset* don't match the clicked position either. They're not quite the same as my calculated one, but only different by a couple of pixels. They're just as far away from the true clicked point as my calculated values.

But in Chrome the coordinates found with offset* are perfect.

Here's a jsfiddle.

Is there anything I'm doing wrong with my calculation? Is there a way for me to mimic Chrome's result, but without the offset* properties?

Answer 1

I am not entirely sure about the theory, I read about this while I was working on a similar problem, but here is what I found. The multiplication of the matrices (the matrix of the 3d transformation - homogeneous coordinates, and the position of the cursor) produces two more values apart from x and y. The first is z, and the other one is w which is used in order to project the 3d object on a 2d plane.

If you divide the x and y values of the vector by the w coordinate, you get the correct position/mapping of the cursor on the Cartesian plane.

So, I would replace the code in your fiddle, lines 69-70, with these:

var x = result.get([0]);
var y = result.get([1]);
var w = result.get([3]);
screenCrosshair.style.left = x / w + 'px';
screenCrosshair.style.top = y / w + 'px';

Here is the update fiddle: https://jsfiddle.net/x3ruc3tL/1/

And then you don't need the offsetX and offsetY values of the browser in order to find the correct position.

Please read the following articles for more information:

https://en.wikipedia.org/wiki/3D_projection#Perspective_projection http://deltaorange.com/2012/03/08/the-truth-behind-homogenous-coordinates/