Object picking with ray casting

Question

I've been having a problem with inaccuracies in my ray casting algorithm for detecting mouse hits within a box. I'm completely at a loss as to how to fix this properly and it's been bugging me for weeks.

The problem is easiest described with a picture (box centered around [0, 0, -30]):

The black lines represent the actual hitbox which is drawn and the green box represents what actually appears to get hit. Notice how it's offset (which seems to get larger if the box is further from the origin) and is slightly smaller than the drawn hitbox.

Here's some relevant code,

ray-box cast:

double BBox::checkFaceIntersection(Vector3 points[4], Vector3 normal, Ray3 ray) {

    double rayDotNorm = ray.direction.dot(normal);
    if(rayDotNorm == 0) return -1;

    Vector3 intersect = points[0] - ray.origin;
    double t = intersect.dot(normal) / rayDotNorm;
    if(t < 0) return -1;

    // Check if first point is from under or below polygon
    bool positive = false;
    double firstPtDot = ray.direction.dot( (ray.origin - points[0]).cross(ray.origin - points[1]) );
    if(firstPtDot > 0) positive = true;
    else if(firstPtDot < 0) positive = false;
    else return -1;

    // Check all signs are the same
    for(int i = 1; i < 4; i++) {
        int nextPoint = (i+1) % 4;
        double rayDotPt = ray.direction.dot( (ray.origin - points[i]).cross(ray.origin - points[nextPoint]) );
        if(positive && rayDotPt < 0) {
            return -1;
        }
        else if(!positive && rayDotPt > 0) {
            return -1;
        }
    }

    return t;
}

mouse to ray:

GLint viewport[4];
GLdouble modelMatrix[16];
GLdouble projectionMatrix[16];

glGetIntegerv(GL_VIEWPORT, viewport);
glGetDoublev(GL_MODELVIEW_MATRIX, modelMatrix);
glGetDoublev(GL_PROJECTION_MATRIX, projectionMatrix);

GLfloat winY = GLfloat(viewport[3] - mouse_y);

Ray3 ray;
double x, y, z;
gluUnProject( (double) mouse_x, winY, 0.0f, // Near
              modelMatrix, projectionMatrix, viewport,
              &x, &y, &z );
ray.origin = Vector3(x, y, z);

gluUnProject( (double) mouse_x, winY, 1.0f, // Far
              modelMatrix, projectionMatrix, viewport,
          &x, &y, &z );
ray.direction = Vector3(x, y, z);

if(bbox.checkBoxIntersection(ray) != -1) {
    std::cout << "Hit!" << std::endl;
}

I've tried drawing the actual ray as a line and it seems to intersect the drawn box correctly.

I had the offset problem partially fixed by minusing all the points and the ray origin/direction by the boxes position, but I have no idea why that worked and the size of the hitbox still remained inaccurate.

Any ideas/alternative approaches? I have other code to supply if it's needed.

Answer 1

You're assuming a wrong direction. Correct would be:

ray.direction = Vector3(far.x - near.x, far.y - near.y, far.z - near.z);

Without subtracting near and far intersection points, your direction will be off.