JavaFX Moving 3D objects with mouse on a virtual plane

Question

As I was creating my first 3D game in JavaFX - where you would be able to assemble ships from parts using the mouse. This presents a problem since JAVAFX seems to have no native metods that work for converting PerspectiveCamera screen 2D coordinates into the scene's 3D space.

Here is a representation of what I'm trying to acheive. A block moved by the mouse should move on an imaginary plane that is always rotated 90 in relation to the camera: I've tried to solve the problem with trigonometry without much success. I've not attached a code snippet as I'm looking for a more generic mathematical solution, but will provide it if requested.

All help would be appreciated!

Desired result:

Answer 1

As @jdub1581 points out, the Camera is the key to bind mouse movement with your 3D objets on the scene.

For starterts, we know about public API PickResult, that allows us to select a 3D object with the mouse, based on some ray tracing techniques performed from the camera position.

But once we have an object, moving it is a different problem.

Looking for a solution for this problem (moving 3D objects with a 2D mouse in 3D space) a while ago, I found the Camera3D class at the Toys project on the OpenJFX repository.

It has a promissing method called unProjectDirection:

/*
 * returns 3D direction from the Camera position to the mouse
 * in the Scene space 
 */

public Vec3d unProjectDirection(double sceneX, double sceneY, 
                                double sWidth, double sHeight) {
}

Since you asked for mathematical explanation, this method uses the trigonometry you were looking for. This will give you a 3D vector based on (x,y) mouse coordinates, using a private Vec3d class (that we can replace with public Point3D):

double tanOfHalfFOV = Math.tan(Math.toRadians(camera.getFieldOfView()) * 0.5f);
Vec3d vMouse = new Vec3d(tanOfHalfFOV*(2*sceneX/sWidth-1), tanOfHalfFOV*(2*sceneY/sWidth-sHeight/sWidth), 1);

Some further transformations are applied to get a normalized vector in scene coordinates.

The next step will transform this normalized vector in real coordinates, just using the distance from the camera to the object, given on the pick result, and transforming the object position.

Basically, this snippet of code outlines the whole process of dragging an object:

    scene.setOnMousePressed((MouseEvent me) -> {
        vecIni = unProjectDirection(me.getSceneX(), me.getSceneY(), 
                     scene.getWidth(),scene.getHeight()); 
        distance=me.getPickResult().getIntersectedDistance();           
    });

    scene.setOnMouseDragged((MouseEvent me) -> {
        vecPos = unProjectDirection(mousePosX, mousePosY,
                 scene.getWidth(),scene.getHeight());
        Point3D p=vecPos.subtract(vecIni).multiply(distance);
        node.getTransforms().add(new Translate(p.getX(),p.getY(),p.getZ()));
        vecIni=vecPos;
        distance=me.getPickResult().getIntersectedDistance();
    });

And this is a full working basic example:

    public class Drag3DObject extends Application {

    private final Group root = new Group();
    private PerspectiveCamera camera;
    private final double sceneWidth = 800;
    private final double sceneHeight = 600;

    private double mousePosX;
    private double mousePosY;
    private double mouseOldX;
    private double mouseOldY;
    private final Rotate rotateX = new Rotate(-20, Rotate.X_AXIS);
    private final Rotate rotateY = new Rotate(-20, Rotate.Y_AXIS);

    private volatile boolean isPicking=false;
    private Point3D vecIni, vecPos;
    private double distance;
    private Sphere s;

    @Override
    public void start(Stage stage) {
        Box floor = new Box(1500, 10, 1500);
        floor.setMaterial(new PhongMaterial(Color.GRAY));
        floor.setTranslateY(150);
        root.getChildren().add(floor);

        Sphere sphere = new Sphere(150);
        sphere.setMaterial(new PhongMaterial(Color.RED));
        sphere.setTranslateY(-5);
        root.getChildren().add(sphere);

        Scene scene = new Scene(root, sceneWidth, sceneHeight, true, SceneAntialiasing.BALANCED);
        scene.setFill(Color.web("3d3d3d"));

        camera = new PerspectiveCamera(true);
        camera.setVerticalFieldOfView(false);

        camera.setNearClip(0.1);
        camera.setFarClip(100000.0);
        camera.getTransforms().addAll (rotateX, rotateY, new Translate(0, 0, -3000));

        PointLight light = new PointLight(Color.GAINSBORO);
        root.getChildren().add(light);
        root.getChildren().add(new AmbientLight(Color.WHITE));
        scene.setCamera(camera);

        scene.setOnMousePressed((MouseEvent me) -> {
            mousePosX = me.getSceneX();
            mousePosY = me.getSceneY();
            PickResult pr = me.getPickResult();
            if(pr!=null && pr.getIntersectedNode() != null && pr.getIntersectedNode() instanceof Sphere){
                distance=pr.getIntersectedDistance();
                s = (Sphere) pr.getIntersectedNode();
                isPicking=true;
                vecIni = unProjectDirection(mousePosX, mousePosY, scene.getWidth(),scene.getHeight());
            }
        });
        scene.setOnMouseDragged((MouseEvent me) -> {
            mousePosX = me.getSceneX();
            mousePosY = me.getSceneY();
            if(isPicking){
                vecPos = unProjectDirection(mousePosX, mousePosY, scene.getWidth(),scene.getHeight());
                Point3D p=vecPos.subtract(vecIni).multiply(distance);
                s.getTransforms().add(new Translate(p.getX(),p.getY(),p.getZ()));
                vecIni=vecPos;
                PickResult pr = me.getPickResult();
                if(pr!=null && pr.getIntersectedNode() != null && pr.getIntersectedNode()==s){
                    distance=pr.getIntersectedDistance();
                } else {
                    isPicking=false;
                }
            } else {
                rotateX.setAngle(rotateX.getAngle()-(mousePosY - mouseOldY));
                rotateY.setAngle(rotateY.getAngle()+(mousePosX - mouseOldX));
                mouseOldX = mousePosX;
                mouseOldY = mousePosY;
            }
        });
        scene.setOnMouseReleased((MouseEvent me)->{
            if(isPicking){
                isPicking=false;
            }
        });

        stage.setTitle("3D Dragging");
        stage.setScene(scene);
        stage.show();
    }

    /*
     From fx83dfeatures.Camera3D
     http://hg.openjdk.java.net/openjfx/8u-dev/rt/file/5d371a34ddf1/apps/toys/FX8-3DFeatures/src/fx83dfeatures/Camera3D.java
    */
    public Point3D unProjectDirection(double sceneX, double sceneY, double sWidth, double sHeight) {
        double tanHFov = Math.tan(Math.toRadians(camera.getFieldOfView()) * 0.5f);
        Point3D vMouse = new Point3D(tanHFov*(2*sceneX/sWidth-1), tanHFov*(2*sceneY/sWidth-sHeight/sWidth), 1);

        Point3D result = localToSceneDirection(vMouse);
        return result.normalize();
    }

    public Point3D localToScene(Point3D pt) {
        Point3D res = camera.localToParentTransformProperty().get().transform(pt);
        if (camera.getParent() != null) {
            res = camera.getParent().localToSceneTransformProperty().get().transform(res);
        }
        return res;
    }

    public Point3D localToSceneDirection(Point3D dir) {
        Point3D res = localToScene(dir);
        return res.subtract(localToScene(new Point3D(0, 0, 0)));
    }

    public static void main(String[] args) {
        launch(args);
    }
}

That will allow you picking and dragging the sphere on the scene: