raycasting: how to properly apply a projection matrix?

Question

I am currently working on some raycasting in GLSL which works fine. Anyways I want to go from orthogonal projection to perspective projection now but I am not sure how to properly do so. Are there any good links on how to use a projection Matrix with raycasting? I am not even sure what I have to apply the matrix to (propably to the ray direction somehow?). Right now I do it like this (pseudocode):

vec3 rayDir = (0.0, 0.0, -1.0); //down the negative -z axis in parallel;

but now I would like to use a projMatrix which works similar to gluPerspective function so that I can simply define an aspect ratio, fov and near and far plane. So basically, can anybody provide me a chunk of code to set up a proj matrix similar to gluProjection does? And secondly tell me if it is correct to multiply it with the rayDirection?

Answer 1

For raytracing in the same scene as a standard render, I have found that the following works for getting a scene-space ray from screen coordinates: (e.g. render a full-screen quad from [-1,-1] to [1,1], or some sub-area within that range)

Vertex Shader

uniform mat4 invprojview;
uniform float near;
uniform float far;

attribute vec2 pos; // from [-1,-1] to [1,1]

varying lowp vec3 origin;
varying lowp vec3 ray;

void main() {
    gl_Position = vec4(pos, 0.0, 1.0);
    origin = (invprojview * vec4(pos, -1.0, 1.0) * near).xyz;
    ray = (invprojview * vec4(pos * (far - near), far + near, far - near)).xyz;

    // equivalent calculation:
    // ray = (invprojview * (vec4(pos, 1.0, 1.0) * far - vec4(pos, -1.0, 1.0) * near)).xyz
}

Fragment Shader

varying lowp vec3 origin;
varying lowp vec3 ray;

void main() {
    lowp vec3 rayDir = normalize(ray);
    // Do raytracing from origin in direction rayDir
}

Note that you need to provide the inverted projection-view matrix, as well as the near and far clipping distances. I'm sure there's a way to get those clipping distances from the matrix, but I haven't figured out how.

This will define a ray which starts at the near plane, not the camera's position. This gives the advantage of clipping at the same position that OpenGL will clip triangles, making your ray-traced object match the scene. Since the ray variable will be the correct length to reach the far plane, you can also clip there too.

As for getting a perspective matrix in the first place (and understanding the mathematics behind it), I always use this reference page:

http://www.songho.ca/opengl/gl_projectionmatrix.html

I recommend looking through the derivation on that site, but in case it becomes unavailable here is the final projection matrix definition:

2n/(r-l)      0      (r+l)/(r-l)      0
    0     2n/(t-b)   (t+b)/(t-b)      0
    0         0     -(f+n)/(f-n)  -2fn/(f-n)
    0         0          -1           0