How to get object in WebGL 3d space from a mouse click coordinate

Question

I'm building a boardgame in WebGL. The board can be rotated/zoomed. I need a way to translate a click on the canvas element (x,y) into the relevant point in 3D space (x, y, z). The ultimate result is that I want to know the (x, y, z) coordinate that contains the point that touches the object closest to the user. For instance, the user clicks a piece, and you imagine a ray traveling through 3D space that goes through both the piece and the game board, but I want the (x, y, z) coord of the piece at the point where it was touched.

I feel like this must be a very common problem, but I can't seem to find a solution in my googles. There must be some way to project the current view of the 3D space into 2D so you can map each point in 2D space to the relevant point in 3D space. I want to the user to be able to mouse over a space on the board, and have the spot change color.

Answer 1

You're looking for an unproject function, which converts screen coordinates into a ray cast from the camera position into the 3D world. You must then perform ray/triangle intersection tests to find the closest triangle to the camera which also intersects the ray.

I have an example of unprojecting available at jax/camera.js#L568 -- but you'll still need to implement ray/triangle intersection. I have an implementation of that at jax/triangle.js#L113.

There is a simpler and (usually) faster alternative, however, called 'picking'. Use this if you want to select an entire object (for instance, a chess piece), and if you don't care about where the mouse actually clicked. The WebGL way to do this is to render the entire scene in various shades of blue (the blue is a key, while red and green are used for unique IDs of the objects in the scene) to a texture, then read back a pixel from that texture. Decoding the RGB into the object's ID will give you the object that was clicked. Again, I've implemented this and it's available at jax/world.js#L82. (See also lines 146, 162, 175.)

Both approaches have pros and cons (discussed here and in some of the comments after) and you'll need to figure out which approach best serves your needs. Picking is slower with huge scenes, but unprojecting in pure JS is extremely slow (since JS itself isn't all that fast) so my best recommendation would be to experiment with both.

FYI, you could also look at the GLU project and unproject code, which I based my code loosely upon: http://www.opengl.org/wiki/GluProject_and_gluUnProject_code

Answer 2

I'm working on this problem at the moment - the approach I'm taking is

1. Render objects to pick buffer each with unique colour
2. Read buffer pixel, map back to picked object
3. Render picked object to buffer with each pixel colour a function of Z-depth
4. Read buffer pixel, map back to Z-depth
5. We have picked object and approximate Z for the pick coords

Answer 3

This is the working demo

function onMouseUp(event) {

    event.preventDefault();        
    x_pos = (event.clientX / window.innerWidth) * 2 - 1;
    y_pos = -(event.clientY / window.innerHeight) * 2 + 1;
    z_pos = 0.5;

    var vector = new THREE.Vector3( x_pos , y_pos , z_pos );

    var projector = new THREE.Projector();
    projector.unprojectVector(vector, camera);
    var raycaster = new THREE.Raycaster(camera.position, vector.sub(camera.position).normalize());
    var intersects = raycaster.intersectObjects(intersectObjects);

    if (intersects.length > 0) {

        xp = intersects[0].point.x.toFixed(2);
        yp = intersects[0].point.y.toFixed(2);
        zp = intersects[0].point.z.toFixed(2);  
        destination = new THREE.Vector3( xp , yp , zp );

        radians =  Math.atan2( ( driller.position.x - xp) , (driller.position.z - zp));
        radians += 90 * (Math.PI / 180);
        console.log(radians);

        var tween = new TWEEN.Tween(driller.rotation).to({ y : radians },200).easing(TWEEN.Easing.Linear.None).start();

    }

weissner-doors.de/drone/