3D Ray Picking use Mouse Coordinates when Mouse isn't locked

Question

So basically I've made a program using OpenGL which can perform 3D Ray Picking. If the Camera View Direction Ray touches/intersects anything (which isn't air) then a little purple box will get rendered at the intersection point/points.

If the ray intersects with any of the "Red Boxes" the once that are intersecting with the ray will turn green. The ground and walls won't change color or texture at all.

Examples:

The current way I do the 3D Ray Picking, is getting the camera's view direction ray, and then just calculating for intersections. My function for calculating intersections doesn't return as a boolean, it returns as a 3D Vector (the coordinates of the intersection itself)

The Question

So what I'm trying to achieve is to calculate the Picking Ray, but according to the mouse when it's not locked to the screen.

Example - So here you can see the purple box is at the crosshair, though if I were to unlocked the mouse and move it (on top of the screen, like normally) and move it over to the center of the green X mark I've draw, then I want to calculate the ray from the camera center to the mouse coordinate on top of the screen.

Current Tests and Ideas

This should just be a mathematical problem. Here is just a short list of the things I currently use to calculate the Ray (and trying to calculate the second ray)

• Cameara X, Y, Z
• Camera Pitch Yaw Roll (Roll not used right now)
• Camera Near Far (distances)
• Camera Fov
• Camera Aspect
• Mouse X, Y (On top of the screen itself)
• Sceen Width, Height

The Mouse X & Y origin (0x0) is in the bottom left corner of the window/frame.

Calculating the main Picking Ray itself

Vector3D position = new Vector3D(
        camera.x,
        camera.y,
        camera.z);

Vector3D direction = new Vector3D(
        Math.cos(Math.toRadians(camera.pitch)) * -Math.sin(Math.toRadians(-camera.yaw)) * camera.far,
        Math.cos(Math.toRadians(camera.pitch)) * cameara.far,
        Math.cos(Math.toRadians(camera.pitch)) * -Math.sin(Math.toRadians(-camera.yaw)) * camera.far);

direction.normalize();

Ray3D ray = new Ray(position, direction);

That is how I calculate the main picking ray itself (the picking ray for the locked mouse). I made the classes myself, though they should make sense (Vector3D, Ray3D, etc) and the normalize() methods does exactly what it says, normalizes the vector.

Idea

So when I tried to calculate using the mouse coordinates I inserted the following code right before I'm calling direction.normalize();, so right after creating the Vector3D direction.

if (!Mouse.isGrabbed())
{
    float mx = Mouse.getX() / (float) scene.width - 0.5f;
    float my = Mouse.getY() / (float) scene.height - 0.5f;

    mx *= camera.far;
    my *= camera.far;

    line.b.x += mx;
    line.b.y += my;
    line.b.z += mz;
}

That gives me a weird result, when the mouse isn't locked/grabbed. It makes sense since I was just messing around and trying some of the things that came first in my mind.

I'm guessing that I need to translate the mouse coordinates according to the pitch, yaw and roll. Though I don't have any idea, how I would go about doing that.

So I'm hoping that there is somebody that can help me achieving this and/or give me some sort of resource so I can understand how to do what I'm trying to do.

Extra

If you need more info about this, just write a comment and I will do my best.

Answer - Thanks to fen

I ended up using fen's way for now, since it was a lot simpler than having to calculate everything!

FloatBuffer projection = BufferTools.createFloatBuffer(16);
FloatBuffer modelview = BufferTools.createFloatBuffer(16);
IntBuffer viewport = BufferTools.createIntBuffer(16);

glGetFloat(GL_PROJECTION_MATRIX, projection);
glGetFloat(GL_MODELVIEW_MATRIX, modelview);
glGetInteger(GL_VIEWPORT, viewport);

float win_x = Mouse.getX();
float win_y = Mouse.getY();

FloatBuffer position_near = BufferTools.createFloatBuffer(3);
FloatBuffer position_far = BufferTools.createFloatBuffer(3);

gluUnProject(win_x, win_y, 0f, modelview, projection, viewport, position_near);
gluUnProject(win_x, win_y, 1f, modelview, projection, viewport, position_far);

Ray3D ray = new Ray3D(
    new Vector3D(
        position_near.get(0),
        position_near.get(1),
        position_near.get(2)),
    new Vector3D(
        position_far.get(0),
        position_far.get(1),
        position_far.get(2)));

Answer 1

here is my code for creating a mouse ray:

double matModelView[16], matProjection[16]; 
int viewport[4]; 

// get matrix and viewport:
glGetDoublev( GL_MODELVIEW_MATRIX, matModelView ); 
glGetDoublev( GL_PROJECTION_MATRIX, matProjection ); 
glGetIntegerv( GL_VIEWPORT, viewport ); 

// window pos of mouse, Y is inverted on Windows
double winX = (double)mouseX; 
double winY = viewport[3] - (double)mouseY; 

// get point on the 'near' plane (third param is set to 0.0)
gluUnProject(winX, winY, 0.0, matModelView, matProjection, 
         viewport, m_start.x, &m_start.y, &m_start.z); 

// get point on the 'far' plane (third param is set to 1.0)
gluUnProject(winX, winY, 1.0, matModelView, matProjection, 
         viewport, m_end.x, &m_end.y, &m_end.z); 

// now you can create a ray from m_start to m_end

OpenGL 2.0, but hope you get the idea.

Some links: Select + Mouse + OpenGL

Answer 2

All you need to do is shoot a ray out from the camera's origin that passes through the screen space point (x,y). The problem here is that to get from your camera's origin to a point in screen space there are a number of transformations that usually occur (2 matrices and a viewport mapping in fact). The other problem is that this throws everything on its head, usually you start with a world space position and wind up with screen space in the OpenGL pipeline -- you want to go the other way :)

You cannot solve this problem with the camera's orientation alone. You need to know how the scene is projected onto your viewing plane, hence the need for the projection matrix. You also need to know the viewport dimensions and the camera's origin. The entire problem can be solved if you know the viewport dimensions, the projection matrix and the modelview matrix.

I would suggest you look into gluUnProject (...), it does everything you need. A quick search on Google resulted in this, which looks pretty helpful: http://myweb.lmu.edu/dondi/share/cg/unproject-explained.pdf