Raycast in Three.js with only a projection matrix

Question

I'm rendering some custom layers using Three.js in a Mapbox GL JS page following this example. I'd like to add raycasting to determine which object a user has clicked on.

The issue is that I only get a projection matrix from Mapbox, which I use to render the scene:

class CustomLayer {
  type = 'custom';
  renderingMode = '3d';

  onAdd(map, gl) {
    this.map = map;
    this.camera = new THREE.Camera();
    this.renderer = new THREE.WebGLRenderer({
      canvas: map.getCanvas(),
      context: gl,
      antialias: true,
    });
    this.scene = new THREE.Scene();
    // ...
  }

  render(gl, matrix) {
    this.camera.projectionMatrix = new THREE.Matrix4()
      .fromArray(matrix)
      .multiply(this.cameraTransform);
    this.renderer.state.reset();
    this.renderer.render(this.scene, this.camera);
  }
}

This renders just great, and tracks changes in view when I pan/rotate/zoom the map.

Unfortunately, when I try to add raycasting I get an error:

  raycast(point) {
    var mouse = new THREE.Vector2();
    mouse.x = ( point.x / this.map.transform.width ) * 2 - 1;
    mouse.y = 1 - ( point.y / this.map.transform.height ) * 2;
    const raycaster = new THREE.Raycaster();
    raycaster.setFromCamera(mouse, this.camera);
    console.log(raycaster.intersectObjects(this.scene.children, true));
  }

This gives me an exception:

THREE.Raycaster: Unsupported camera type.

I can change from a generic THREE.Camera to a THREE.PerspectiveCamera without affecting the rendering of the scene:

this.camera = new THREE.PerspectiveCamera(28, window.innerWidth / window.innerHeight, 0.1, 1e6);

This fixes the exception but also doesn't result in any objects being logged. Digging a bit reveals that the camera's projectionMatrixInverse is all NaNs, which we can fix by calculating it:

  raycast(point) {
    var mouse = new THREE.Vector2();
    mouse.x = ( point.x / this.map.transform.width ) * 2 - 1;
    mouse.y = 1 - ( point.y / this.map.transform.height ) * 2;
    this.camera.projectionMatrixInverse.getInverse(this.camera.projectionMatrix);  // <--
    const raycaster = new THREE.Raycaster();
    raycaster.setFromCamera(mouse, this.camera);
    console.log(raycaster.intersectObjects(this.scene.children, true));
  }

Now I get two intersections wherever I click, with two faces of the cube. Their distances are 0:

[
  { distance: 0, faceIndex: 10, point: Vector3 { x: 0, y: 0, z: 0 }, uv: Vector2 {x: 0.5, y: 0.5}, ... },
  { distance: 0, faceIndex: 11, point: Vector3 { x: 0, y: 0, z: 0 }, uv: Vector2 {x: 0.5, y: 0.5}, ... },
]

So clearly something isn't working here. Looking at the code for setCamera, it involves both projectionMatrix and matrixWorld. Is there a way I can set matrixWorld, or construct the raycaster's ray directly using only the projection matrix? It seems that I only need the projection matrix to render the scene, so I'd hope that it would also be all I need to cast a ray.

Full example in this codepen.

Answer 1

Difficult to debug, but solved!

TL;DR: Full code in this working Fiddle.

1. I think that not having the world matrix of the camera from MapBox is not the main problem, rather the incompatibility of the coordinate spaces. Mapbox delivers a left-handed system with z pointing up. Three uses a right-handed system with y pointing up.

2. During the debugging I created a reduced copy of the raycaster setup functions to have everything under control and it paid off.

3. A cube is not the best object for debugging, It is way too symmetric. The best are asymmetric primitives or compound objects. I prefer a 3D coordinate cross to see the axes orientation straight away.

Coordinate systems

• There is no need for changing the DefaultUp in the BoxCustomLayer constructor, since the goals is to have everything aligned with the default coordinates in Three.
• You flip the y-axis in the onAdd() method, but then you also scale all objects when initializing the group. This way, the coordinates you operate in after inverting the projection in raycast() are not in meters anymore. So let's join that scaling part with this.cameraTransform. The left-handed to right-handed conversion should be done here as well. I decided to flip the z-axis and rotate 90deg along x-axis to get a right-handed system with y pointing up:

const centerLngLat = map.getCenter();
this.center = MercatorCoordinate.fromLngLat(centerLngLat, 0);
const {x, y, z} = this.center;

const s = this.center.meterInMercatorCoordinateUnits();

const scale = new THREE.Matrix4().makeScale(s, s, -s);
const rotation = new THREE.Matrix4().multiplyMatrices(
        new THREE.Matrix4().makeRotationX(-0.5 * Math.PI),
        new THREE.Matrix4().makeRotationY(Math.PI)); //optional Y rotation

this.cameraTransform = new THREE.Matrix4()
        .multiplyMatrices(scale, rotation)
        .setPosition(x, y, z);

3. Make sure to remove the scaling from the group in makeScene , in fact you don't need the group anymore.

4. No need to touch the render function.

Raycasting

Here it gets a bit tricky, basically it's what Raycaster would do, but I left out some unnecessary function calls, e.g. the camera world matrix is identity => no need to multiply with it.

1. Get the inverse of the projection matrix. It does not update when Object3D.projectionMatrix is assigned, so let's compute it manually.
2. Use it to get the camera and mouse position in 3D. This is equivalent to Vector3.unproject.
3. viewDirection is simply the normalized vector from the cameraPosition to the mousePosition.

const camInverseProjection = 
        new THREE.Matrix4().getInverse(this.camera.projectionMatrix);
const cameraPosition =
        new THREE.Vector3().applyMatrix4(camInverseProjection);
const mousePosition =
        new THREE.Vector3(mouse.x, mouse.y, 1)
        .applyMatrix4(camInverseProjection);
const viewDirection = mousePosition.clone()
        .sub(cameraPosition).normalize();

4. Setup the raycaster ray

this.raycaster.set(cameraPosition, viewDirection);

5. The rest can stay the same.

Full code

Fiddle for demonstration.

• I added mousemove to display real-time debug info with jQuery.
• The full info about the hit gets logged to the console upon click.