WebXR controllers for button pressing in three.js

Question

I would like to figure out how to map out the controls for my oculus quest and other devices, using three.js and webXR. The code works, and allows me to move the controller, maps a cylinder to each control, and allows me to use the trigger to controls to change the color of the cylinders. This is great, but I can't find any documentation on how to use axis controls for the joy stick, the grip and the other buttons. Part of me wants to believe it's as simple as knowing which event to call, because I don't know what other events are available.

Here is a link to the tutorial I based this off of. https://github.com/as-ideas/webvr-with-threejs

Please note that this code works as expected, but I don't know how totake it further and do more.

function createController(controllerID, videoinput) { 
//RENDER CONTROLLER AS YELLOW TUBE
        const controller = renderer.vr.getController(controllerID);
        const cylinderGeometry = new CylinderGeometry(0.025, 0.025, 1, 32);
        const cylinderMaterial = new MeshPhongMaterial({ color: 0xffff00 });
        const cylinder = new Mesh(cylinderGeometry, cylinderMaterial);
        cylinder.geometry.translate(0, 0.5, 0);
        cylinder.rotateX(-0.25 * Math.PI);
        controller.add(cylinder);
        cameraFixture.add(controller);
        //TRIGGER
        controller.addEventListener('selectstart', () => {
            if (controllerID === 0) {
                cylinderMaterial.color.set('pink')
            } else {
                cylinderMaterial.color.set('orange');
                videoinput.play()
            }
        });
        controller.addEventListener('selectend', () => {
            cylinderMaterial.color.set(0xffff00);
            videoinput.pause();
            console.log('I pressed play');
        });
    }

Answer 1

As of three.js 0.119, integrated 'events' from the other buttons, trackpads, haptics, and thumbsticks of a touch controller are not provided, only select and squeeze events are available. three.js has a functional model of 'just working' regardless of what type of input device you have and only provides for managing events that can be produced by all input devices (ie. select)

Luckily, we are not limited by what three.js has made available and can just poll the controller data directly.

---

Touch controllers follow the model of 'gamepad' controls and just report their instantanous values. We will poll the gamepad for its current values of the various buttons and keep track of their state and create 'events' within our code for button pushes, trackpad and thumbstick axis changes.

To access the instantaneous data from a touch controller while within a webXR session

const session = renderer.xr.getSession();
let i = 0;

if (session) {
        for (const source of session.inputSources) {
            if (source && source.handedness) {
                handedness = source.handedness; //left or right controllers
            }
            if (!source.gamepad) continue;
            const controller = renderer.xr.getController(i++);
            const old = prevGamePads.get(source);
            const data = {
                handedness: handedness,
                buttons: source.gamepad.buttons.map((b) => b.value),
                axes: source.gamepad.axes.slice(0)
            };
            //process data accordingly to create 'events'

---

Haptic feedback is provided through a promise (Note not all browsers currently support the webXR haptic feedback, but Oculus Browser and Firefox Reality on quest do) When availble, the haptic feedback is produced through a promise:

var didPulse = sourceXR.gamepad.hapticActuators[0].pulse(0.8, 100);
//80% intensity for 100ms
//subsequent promises cancel any previous promise still underway

---

To demonstrate this solution I have modified threejs.org/examples/#webXR_vr_dragging example by adding the camera to a 'dolly' that can be moved around with the touch controllers thumbsticks when within a webXR session and provide various haptic feedback for events such as raycasting onto an object or axis movements on thumbsticks.

For each frame, we poll the data from the touch controllers and respond accordingly. We have to store the data from frame to frame to detect changes and create our events, and filter out some data (false 0's and up to 20% randomdrift from 0 in thumbstick axis values on some controllers) For proper 'forward and sideways' dolly movement the current heading and attitude of the webXR camera is also needed each frame and accessed via:

    let xrCamera = renderer.xr.getCamera(camera);
    xrCamera.getWorldDirection(cameraVector);
    //heading vector for webXR camera now within cameraVector

---

Example codepen here: codepen.io/jason-buchheim/pen/zYqYGXM

With 'ENTER VR' button exposed (debug view) here:cdpn.io/jason-buchheim/debug/zYqYGXM

---

Full code with modifications of original threejs example highlighted with comment blocks

//// From webxr_vr_dragging example https://threejs.org/examples/#webxr_vr_dragging
import * as THREE from "https://cdn.jsdelivr.net/npm/[email protected]/build/three.module.min.js";
import { OrbitControls } from "https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/controls/OrbitControls.min.js";
import { VRButton } from "https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/webxr/VRButton.min.js";
import { XRControllerModelFactory } from "https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/webxr/XRControllerModelFactory.min.js";

var container;
var camera, scene, renderer;
var controller1, controller2;
var controllerGrip1, controllerGrip2;

var raycaster,
    intersected = [];
var tempMatrix = new THREE.Matrix4();

var controls, group;

////////////////////////////////////////
//// MODIFICATIONS FROM THREEJS EXAMPLE
//// a camera dolly to move camera within webXR
//// a vector to reuse each frame to store webXR camera heading
//// a variable to store previous frames polling of gamepads
//// a variable to store accumulated accelerations along axis with continuous movement

var dolly;
var cameraVector = new THREE.Vector3(); // create once and reuse it!
const prevGamePads = new Map();
var speedFactor = [0.1, 0.1, 0.1, 0.1];

////
//////////////////////////////////////////
init();
animate();

function init() {
    container = document.createElement("div");
    document.body.appendChild(container);

    scene = new THREE.Scene();
    scene.background = new THREE.Color(0x808080);

    camera = new THREE.PerspectiveCamera(
        50,
        window.innerWidth / window.innerHeight,
        0.1,
        500  //MODIFIED FOR LARGER SCENE
    );
    camera.position.set(0, 1.6, 3);

    controls = new OrbitControls(camera, container);
    controls.target.set(0, 1.6, 0);
    controls.update();

    var geometry = new THREE.PlaneBufferGeometry(100, 100);
    var material = new THREE.MeshStandardMaterial({
        color: 0xeeeeee,
        roughness: 1.0,
        metalness: 0.0
    });
    var floor = new THREE.Mesh(geometry, material);
    floor.rotation.x = -Math.PI / 2;
    floor.receiveShadow = true;
    scene.add(floor);

    scene.add(new THREE.HemisphereLight(0x808080, 0x606060));

    var light = new THREE.DirectionalLight(0xffffff);
    light.position.set(0, 200, 0);           // MODIFIED SIZE OF SCENE AND SHADOW
    light.castShadow = true;
    light.shadow.camera.top = 200;           // MODIFIED FOR LARGER SCENE
    light.shadow.camera.bottom = -200;       // MODIFIED FOR LARGER SCENE
    light.shadow.camera.right = 200;         // MODIFIED FOR LARGER SCENE
    light.shadow.camera.left = -200;         // MODIFIED FOR LARGER SCENE
    light.shadow.mapSize.set(4096, 4096);
    scene.add(light);

    group = new THREE.Group();
    scene.add(group);

    var geometries = [
        new THREE.BoxBufferGeometry(0.2, 0.2, 0.2),
        new THREE.ConeBufferGeometry(0.2, 0.2, 64),
        new THREE.CylinderBufferGeometry(0.2, 0.2, 0.2, 64),
        new THREE.IcosahedronBufferGeometry(0.2, 3),
        new THREE.TorusBufferGeometry(0.2, 0.04, 64, 32)
    ];

    for (var i = 0; i < 100; i++) {
        var geometry = geometries[Math.floor(Math.random() * geometries.length)];
        var material = new THREE.MeshStandardMaterial({
            color: Math.random() * 0xffffff,
            roughness: 0.7,
            side: THREE.DoubleSide,   // MODIFIED TO DoubleSide
            metalness: 0.0
        });

        var object = new THREE.Mesh(geometry, material);

        object.position.x = Math.random() * 200 - 100;  // MODIFIED FOR LARGER SCENE
        object.position.y = Math.random() * 100;        // MODIFIED FOR LARGER SCENE
        object.position.z = Math.random() * 200 - 100;  // MODIFIED FOR LARGER SCENE

        object.rotation.x = Math.random() * 2 * Math.PI;
        object.rotation.y = Math.random() * 2 * Math.PI;
        object.rotation.z = Math.random() * 2 * Math.PI;

        object.scale.setScalar(Math.random() * 20 + 0.5);  // MODIFIED FOR LARGER SCENE

        object.castShadow = true;
        object.receiveShadow = true;

        group.add(object);
    }

    // renderer
    renderer = new THREE.WebGLRenderer({ antialias: true });
    renderer.setPixelRatio(window.devicePixelRatio);
    renderer.setSize(window.innerWidth, window.innerHeight);
    renderer.outputEncoding = THREE.sRGBEncoding;
    renderer.shadowMap.enabled = true;
    renderer.xr.enabled = true;
    //the following increases the resolution on Quest
    renderer.xr.setFramebufferScaleFactor(2.0);
    container.appendChild(renderer.domElement);
    document.body.appendChild(VRButton.createButton(renderer));

    // controllers
    controller1 = renderer.xr.getController(0);
    controller1.name="left";    ////MODIFIED, added .name="left"
    controller1.addEventListener("selectstart", onSelectStart);
    controller1.addEventListener("selectend", onSelectEnd);
    scene.add(controller1);

    controller2 = renderer.xr.getController(1);
    controller2.name="right";  ////MODIFIED added .name="right"
    controller2.addEventListener("selectstart", onSelectStart);
    controller2.addEventListener("selectend", onSelectEnd);
    scene.add(controller2);

    var controllerModelFactory = new XRControllerModelFactory();

    controllerGrip1 = renderer.xr.getControllerGrip(0);
    controllerGrip1.add(
        controllerModelFactory.createControllerModel(controllerGrip1)
    );
    scene.add(controllerGrip1);

    controllerGrip2 = renderer.xr.getControllerGrip(1);
    controllerGrip2.add(
        controllerModelFactory.createControllerModel(controllerGrip2)
    );
    scene.add(controllerGrip2);

    //Raycaster Geometry
    var geometry = new THREE.BufferGeometry().setFromPoints([
        new THREE.Vector3(0, 0, 0),
        new THREE.Vector3(0, 0, -1)
    ]);

    var line = new THREE.Line(geometry);
    line.name = "line";
    line.scale.z = 50;   //MODIFIED FOR LARGER SCENE

    controller1.add(line.clone());
    controller2.add(line.clone());

    raycaster = new THREE.Raycaster();

    ////////////////////////////////////////
    //// MODIFICATIONS FROM THREEJS EXAMPLE
    //// create group named 'dolly' and add camera and controllers to it
    //// will move dolly to move camera and controllers in webXR

    dolly = new THREE.Group();
    dolly.position.set(0, 0, 0);
    dolly.name = "dolly";
    scene.add(dolly);
    dolly.add(camera);
    dolly.add(controller1);
    dolly.add(controller2);
    dolly.add(controllerGrip1);
    dolly.add(controllerGrip2);

    ////
    ///////////////////////////////////

    window.addEventListener("resize", onWindowResize, false);
}

function onWindowResize() {
    camera.aspect = window.innerWidth / window.innerHeight;
    camera.updateProjectionMatrix();

    renderer.setSize(window.innerWidth, window.innerHeight);
}

function onSelectStart(event) {
    var controller = event.target;

    var intersections = getIntersections(controller);

    if (intersections.length > 0) {
        var intersection = intersections[0];
        var object = intersection.object;
        object.material.emissive.b = 1;
        controller.attach(object);
        controller.userData.selected = object;
    }
}

function onSelectEnd(event) {
    var controller = event.target;
    if (controller.userData.selected !== undefined) {
        var object = controller.userData.selected;
        object.material.emissive.b = 0;
        group.attach(object);
        controller.userData.selected = undefined;
    }
}

function getIntersections(controller) {
    tempMatrix.identity().extractRotation(controller.matrixWorld);
    raycaster.ray.origin.setFromMatrixPosition(controller.matrixWorld);
    raycaster.ray.direction.set(0, 0, -1).applyMatrix4(tempMatrix);
    return raycaster.intersectObjects(group.children);
}

function intersectObjects(controller) {
    // Do not highlight when already selected

    if (controller.userData.selected !== undefined) return;

    var line = controller.getObjectByName("line");
    var intersections = getIntersections(controller);

    if (intersections.length > 0) {
        var intersection = intersections[0];

        ////////////////////////////////////////
        //// MODIFICATIONS FROM THREEJS EXAMPLE
        //// check if in webXR session
        //// if so, provide haptic feedback to the controller that raycasted onto object
        //// (only if haptic actuator is available)
        const session = renderer.xr.getSession();
        if (session) {  //only if we are in a webXR session
            for (const sourceXR of session.inputSources) {

                if (!sourceXR.gamepad) continue;
                if (
                    sourceXR &&
                    sourceXR.gamepad &&
                    sourceXR.gamepad.hapticActuators &&
                    sourceXR.gamepad.hapticActuators[0] &&
                    sourceXR.handedness == controller.name              
                ) {
                    var didPulse = sourceXR.gamepad.hapticActuators[0].pulse(0.8, 100);
                }
            }
        }
        ////
        ////////////////////////////////

        var object = intersection.object;
        object.material.emissive.r = 1;
        intersected.push(object);

        line.scale.z = intersection.distance;
    } else {
        line.scale.z = 50;   //MODIFIED AS OUR SCENE IS LARGER
    }
}

function cleanIntersected() {
    while (intersected.length) {
        var object = intersected.pop();
        object.material.emissive.r = 0;
    }
}

function animate() {
    renderer.setAnimationLoop(render);
}

function render() {
    cleanIntersected();

    intersectObjects(controller1);
    intersectObjects(controller2);

    ////////////////////////////////////////
    //// MODIFICATIONS FROM THREEJS EXAMPLE

    //add gamepad polling for webxr to renderloop
    dollyMove();

    ////
    //////////////////////////////////////

    renderer.render(scene, camera);
}


////////////////////////////////////////
//// MODIFICATIONS FROM THREEJS EXAMPLE
//// New dollyMove() function
//// this function polls gamepad and keeps track of its state changes to create 'events'

function dollyMove() {
    var handedness = "unknown";

    //determine if we are in an xr session
    const session = renderer.xr.getSession();
    let i = 0;

    if (session) {
        let xrCamera = renderer.xr.getCamera(camera);
        xrCamera.getWorldDirection(cameraVector);

        //a check to prevent console errors if only one input source
        if (isIterable(session.inputSources)) {
            for (const source of session.inputSources) {
                if (source && source.handedness) {
                    handedness = source.handedness; //left or right controllers
                }
                if (!source.gamepad) continue;
                const controller = renderer.xr.getController(i++);
                const old = prevGamePads.get(source);
                const data = {
                    handedness: handedness,
                    buttons: source.gamepad.buttons.map((b) => b.value),
                    axes: source.gamepad.axes.slice(0)
                };
                if (old) {
                    data.buttons.forEach((value, i) => {
                        //handlers for buttons
                        if (value !== old.buttons[i] || Math.abs(value) > 0.8) {
                            //check if it is 'all the way pushed'
                            if (value === 1) {
                                //console.log("Button" + i + "Down");
                                if (data.handedness == "left") {
                                    //console.log("Left Paddle Down");
                                    if (i == 1) {
                                        dolly.rotateY(-THREE.Math.degToRad(1));
                                    }
                                    if (i == 3) {
                                        //reset teleport to home position
                                        dolly.position.x = 0;
                                        dolly.position.y = 5;
                                        dolly.position.z = 0;
                                    }
                                } else {
                                    //console.log("Right Paddle Down");
                                    if (i == 1) {
                                        dolly.rotateY(THREE.Math.degToRad(1));
                                    }
                                }
                            } else {
                                // console.log("Button" + i + "Up");

                                if (i == 1) {
                                    //use the paddle buttons to rotate
                                    if (data.handedness == "left") {
                                        //console.log("Left Paddle Down");
                                        dolly.rotateY(-THREE.Math.degToRad(Math.abs(value)));
                                    } else {
                                        //console.log("Right Paddle Down");
                                        dolly.rotateY(THREE.Math.degToRad(Math.abs(value)));
                                    }
                                }
                            }
                        }
                    });
                    data.axes.forEach((value, i) => {
                        //handlers for thumbsticks
                        //if thumbstick axis has moved beyond the minimum threshold from center, windows mixed reality seems to wander up to about .17 with no input
                        if (Math.abs(value) > 0.2) {
                            //set the speedFactor per axis, with acceleration when holding above threshold, up to a max speed
                            speedFactor[i] > 1 ? (speedFactor[i] = 1) : (speedFactor[i] *= 1.001);
                            console.log(value, speedFactor[i], i);
                            if (i == 2) {
                                //left and right axis on thumbsticks
                                if (data.handedness == "left") {
                                    // (data.axes[2] > 0) ? console.log('left on left thumbstick') : console.log('right on left thumbstick')

                                    //move our dolly
                                    //we reverse the vectors 90degrees so we can do straffing side to side movement
                                    dolly.position.x -= cameraVector.z * speedFactor[i] * data.axes[2];
                                    dolly.position.z += cameraVector.x * speedFactor[i] * data.axes[2];

                                    //provide haptic feedback if available in browser
                                    if (
                                        source.gamepad.hapticActuators &&
                                        source.gamepad.hapticActuators[0]
                                    ) {
                                        var pulseStrength = Math.abs(data.axes[2]) + Math.abs(data.axes[3]);
                                        if (pulseStrength > 0.75) {
                                            pulseStrength = 0.75;
                                        }

                                        var didPulse = source.gamepad.hapticActuators[0].pulse(
                                            pulseStrength,
                                            100
                                        );
                                    }
                                } else {
                                    // (data.axes[2] > 0) ? console.log('left on right thumbstick') : console.log('right on right thumbstick')
                                    dolly.rotateY(-THREE.Math.degToRad(data.axes[2]));
                                }
                                controls.update();
                            }

                            if (i == 3) {
                                //up and down axis on thumbsticks
                                if (data.handedness == "left") {
                                    // (data.axes[3] > 0) ? console.log('up on left thumbstick') : console.log('down on left thumbstick')
                                    dolly.position.y -= speedFactor[i] * data.axes[3];
                                    //provide haptic feedback if available in browser
                                    if (
                                        source.gamepad.hapticActuators &&
                                        source.gamepad.hapticActuators[0]
                                    ) {
                                        var pulseStrength = Math.abs(data.axes[3]);
                                        if (pulseStrength > 0.75) {
                                            pulseStrength = 0.75;
                                        }
                                        var didPulse = source.gamepad.hapticActuators[0].pulse(
                                            pulseStrength,
                                            100
                                        );
                                    }
                                } else {
                                    // (data.axes[3] > 0) ? console.log('up on right thumbstick') : console.log('down on right thumbstick')
                                    dolly.position.x -= cameraVector.x * speedFactor[i] * data.axes[3];
                                    dolly.position.z -= cameraVector.z * speedFactor[i] * data.axes[3];

                                    //provide haptic feedback if available in browser
                                    if (
                                        source.gamepad.hapticActuators &&
                                        source.gamepad.hapticActuators[0]
                                    ) {
                                        var pulseStrength = Math.abs(data.axes[2]) + Math.abs(data.axes[3]);
                                        if (pulseStrength > 0.75) {
                                            pulseStrength = 0.75;
                                        }
                                        var didPulse = source.gamepad.hapticActuators[0].pulse(
                                            pulseStrength,
                                            100
                                        );
                                    }
                                }
                                controls.update();
                            }
                        } else {
                            //axis below threshold - reset the speedFactor if it is greater than zero  or 0.025 but below our threshold
                            if (Math.abs(value) > 0.025) {
                                speedFactor[i] = 0.025;
                            }
                        }
                    });
                }
                ///store this frames data to compate with in the next frame
                prevGamePads.set(source, data);
            }
        }
    }
}

function isIterable(obj) {  //function to check if object is iterable
    // checks for null and undefined
    if (obj == null) {
        return false;
    }
    return typeof obj[Symbol.iterator] === "function";
}

////
/////////////////////////////////////