Translate mouse coordinates to a 3D plan

Question

I'm building some kind of drag and drop app in javascript/jquery (DOM based, not canvas).

The idea is to be able to drag divs on a 3D scene (a div rotated in 3D).

It works on a 2D plan, the problem is when I rotate the scene in 3D, the objects position doesn't reflect the actual mouse position, but the coordinates translated in 3D

Illustrated exemple :

EXEMPLE ON JSFIDDLE

I want the objects moving relative to the absolute position of the mouse.

I calculate the mouse position like this :

document.addEventListener(gestureMove, function (event) {
  if (mouseDown == true) {
  event.preventDefault();
  moveX = (event.pageX - $('#scene').offset().left);
  moveY = (event.pageY - $('#scene').offset().top);
}

#scene { 
  width: 1000px;
  height: 1000px;
  -webkit-transform-style: preserve-3d;
  -webkit-transform: rotateX( 35deg ); 
}

An early solution was to calculate the difference between the mouse position and the object based on the initial position, and adding it to the object position during the drag. It was working, but the animation was really choppy and not smooth at all.

I'm sure there is a more simple way to get the mouse coordinates relative to the 3D plan, but wasn't able to find a real solution at this point.

Most of the search results on this subject point to gaming oriented languages, or canvas/webgl questions.

Any ideas ?

Thanks

Answer 1

Assuming that your mouse position is an absolute screen position, and you want to grab and slide an object around on a 3D plane based directly on the mouse position:

You can represent your 3D target plane as:

• a 3D origin point O
• two 3D vectors U and V, representing the direction of the U- and V-axes

Then, a given 3D point corresponding to plane coordinates [u,v] is:

point3d P = O + u*U + v*V

Then, you can combine the operations that map this particular 3D point to the screen; this is usually described with 3D transformation matrices ModelMatrix, ViewMatrix, and ProjectionMatrix, and a viewport transform determined by a 2D screen origin point origin_2d and a 2D scaling vector scale_2d. To solve the problem in a readily invertible way, promote everything to homogeneous coordinates, by adding a .w-coordinate to each of them. This extra coordinate acts as a scaling factor -- to get cartesian coordinates back, you need to divide the homogeneous .x and .y values by the .w value:

P_hom = [u, v, 1] * [U.x, U.y, U.z, 0] = [u, v, 1] * TexMatrix
                    [V.x, V.y, V.z, 0]
                    [O.x, O.y, O.z, 1]

P_clip_hom = P_hom * ModelMatrix * ViewMatrix * ProjectionMatrix
           = P_hom * ModelViewProjectionMatrix

screenpos_hom = P_clip_hom * [scale_2d.x     0        0] = P_clip_hom * PortMatrix
                             [   0        scale_2d.y  0]
                             [   0           0        0]
                             [origin_2d.x origin_2d.y 1]

So, screenpos_hom = [u, v, 1] * TexMatrix * ModelViewProjectionMatrix * PortMatrix
                  = [u, v, 1] * TexToScreenMatrix

-> [screenpos.x, screenpos.y] = [screenpos_hom.x, screenpos_hom.y] / screenpos_hom.w

Note that TexToScreenMatrix is a 3x3 matrix; you should be able to invert it:

UV_2d_hom = [screenpos.x, screenpos.y, 1] * (TexToScreenMatrix)^-1

-> [u, v] = [UV_2d_hom.x, UV_2d_hom.y] / UV_2d_hom.w

Finally, you can either use the [u,v] coordinates directly, or you can use them to recreate the 3D point P as described above.