Using GLM's UnProject

Question

I'm not sure how to use the Unproject method provided by GLM.

Specifically, in what format is the viewport passed in? And why doesn't the function require a view matrix as well as a projection and world matrix?

Answer 1

A bit of history is required here. GLM's unproject is actually a more or less direct replacement for the gluUnProject function that uses deprecated OpenGL fixed-function rendering. In this mode the Model and View matrix were actually combined in the "ModelView" matrix. Apparently, the GLM author dropped the 'view' part in the naming, which confuses thing even more, but it comes down to passing something like view*model.

Now for the actual use:

• win is a vector that holds three components that have meaning in Window coordinates. These are coordinates 'x,y' in your viewport, the 'z' coordinate you usually retrieve by reading your depth buffer at (x,y)
• model, view and projection matrices should speak for itself if you're even thinking of using this function. But a good (opengl-specific) refresher can be useful.
• The viewport is defined as in glViewport, which means (x,y,w,h). X and Y specify the lower left corner of your viewport (usually 0,0). Width and height (w,h). Note that in many other systems x,y specify the upper left corner, you have to transform your y-coordinate then, which is shown in the NeHe code I link to below.

When applied you simply end up by converting the provided window coordinates back to the object coordinates, more or less the inverse of what your render code usually does.

A half-decent explanation on the original gluUnProject can be found as a NeHe article. But of course that is OpenGL-specific, while glm can be used in other contexts.

Answer 2

The viewport is passed in as four floats: the x and y window coordinates of the viewport, followed by its width and height. That's the same order as used e.g. by glGetFloatv(GL_VIEWPORT, ...). So in most cases, the first two values should be 0.

As KillianDS already pointed out, the modelargument in fact is a modelview matrix, see the example use of unProject() in gtx_simd_mat4.cpp, function test_compute_gtx():

    glm::mat4 E = glm::translate(D, glm::vec3(1.4f, 1.2f, 1.1f));
    glm::mat4 F = glm::perspective(i, 1.5f, 0.1f, 1000.f);
    glm::mat4 G = glm::inverse(F * E);
    glm::vec3 H = glm::unProject(glm::vec3(i), G, F, E[3]);

As you can see, the matrix passed as the second argument basically is the product of a translation and a perspective transformation.