xy position at a certain z depth

Question

In processing (java dialect) there are the methods screenX, screenY (and screenZ but we skip that for now).

Let's say i have a object at xyz = 50, 100, 500. Then with screenX and screenY you can now where they will apear on the canvas.

float x = screenX(50, 100, 500); float y = screenY(50, 100, 500);

here is the reference: http://processing.org/reference/screenX_.html

What i'm interested in is like a inverse method. For example, i want a sphere to apear on the canvas on x=175 and y=100. The sphere should have a z of 700. Then what would the actual x and y position be at z=700 to make it apear on the canvas at 175,100?

So the method would be float unscreenX(float x, float y, float z) and it would return the x value.

My math / programming skills is not so advanced (let's call it bad) (i'm more a designer) so i'm looking for some help. I all ready asked at the processing board but often there are more people here with deeper knowledge about matrixes etc.

The normal screenX method from processing can be found here: https://github.com/processing/processing/blob/master/core/src/processing/opengl/PGraphicsOpenGL.java

public float screenX(float x, float y, float z) {
    return screenXImpl(x, y, z);
  }

protected float screenXImpl(float x, float y, float z) {
    float ax =
      modelview.m00*x + modelview.m01*y + modelview.m02*z + modelview.m03;
    float ay =
      modelview.m10*x + modelview.m11*y + modelview.m12*z + modelview.m13;
    float az =
      modelview.m20*x + modelview.m21*y + modelview.m22*z + modelview.m23;
    float aw =
      modelview.m30*x + modelview.m31*y + modelview.m32*z + modelview.m33;
    return screenXImpl(ax, ay, az, aw);
  }


  protected float screenXImpl(float x, float y, float z, float w) {
    float ox =
      projection.m00*x + projection.m01*y + projection.m02*z + projection.m03*w;
    float ow =
      projection.m30*x + projection.m31*y + projection.m32*z + projection.m33*w;

    if (nonZero(ow)) {
      ox /= ow;
    }
    float sx = width * (1 + ox) / 2.0f;
    return sx;
  }

Of corse there is also for the y and for the z (i don't understand the z but let's ignore that). I thought this might give some insight in how to inverse it.

modelview and projection is a 3d matrix, the code is here: https://github.com/processing/processing/blob/master/core/src/processing/core/PMatrix3D.java But i guess it's pretty basic and common.

I also made a post on the processing board since you never know. It explains a litle bit different what i want. http://forum.processing.org/topic/unscreenx-and-unscreeny

For the tags describing this post, i didn't went to specific cause i can imagine a programmer who never worked with java but did work with c++ for example and has experience in matrixes is still able to provide a good answer.

hope someone can help.

Answer 1

I highly recommend you study some linear algebra or matrix math for 3d graphics. It's fun and easy, but a bit longer than a SO answer. I'll try though :) Disclaimer: I have no idea about the API you are using!

It looks like you are returning 3 coordinate for a position (often called a vertex). But you also mention a projection matrix and that function has 4 coordinates. Usually a shader or API will take 4 coordinates for a vertex. x,y,z,w. To get them on screen it does something like this:

xscreen = x/w
yscreen = y/w
zbuffer = z/w

This is useful because you get to pick w. If you are just doing 2d drawing you can just put w=1. But if you are doing 3d and want some perspective effect you want to divide by distance from the camera. And that's what the projection matrix is for. It mainly takes the z of your point where z means distance to camera and puts it into w. It also might scale things around a bit, like field of view.

Looking back at the code you posted this is exactly what the last ScreenXImpl function does. It applies a projection matrix, which mostly just moves z into w, and then divides by w. At the end it does an extra scale and offset from (-1,1) to (0,widhtinpixels) but we can ignore that.

Now, why am I rambling on about this stuff? All you want to do is to get the x,y,z coordinates for a given xscreen, yscreen, zbuffer, right? Well, the trick is just going backwards. In in order to do that you need to have a firm grasp on going forward :)

There are two problems with going backwards: 1) Do you really know or care for the zbuffer value? 2) Do you know what the projection matrix did? For 1) Let's say we don't care. There's many possible values for that, so we might just pick one. For 2) You will have to look at what it does. Some projection matrices might just take (x,y,z,w) and output (x,y,z,1). That would be 2d. Or (x,y+z,z,1) which would be isometric. But in perspective it usually does (x,y,1,z). Plus some scaling and so on.

I just noticed your second screenXImpl already passes x,y,z,w to the next stage. That is useful sometimes, but for all practical cases that w will be 1.

At this point I realize that I am terrible at explaining things. :) You really should pick up that linear algebra book, I learned from this one: http://www.amazon.com/Elementary-Linear-Algebra-Howard-Anton but it came with a good lecture, so I don't know how useful it is on it's own.

Anyhow! Let's get more practical. Back to your code: the last function of screenXImpl. We now know that the input w=1 and that ow=~z and ox=~x; The squiggly line here means times some scale plus some offset. And the screen x we have to begin with is ~ox/ow. (+1/2,*width.. that's what squiggly lines are for). And now we are back at 1)... if you want a special oz - pick one now. Otherwise, we can pick any. For rendering it probably makes sense to pick anything in front of the camera and easy to work with. Like 1.

protected float screenXImpl(float x, float y, float z, float w==1) {
float ox = 1*x + 0*y + 0*z + 0*w; // == x
float ow = 0*x + 0*y + 1*z + 0*w; // == z == 1

ox /= ow; // == ox

float sx = width * (1 + ox) / 2.0f;
return sx;
}

WTF? sx = width * (1+ox)/2 ? Why didn't I just say so? Well, all the zeros I put in there are probably not zero. But it's going to end up just as simple. Ones might not be ones. I tried to show the important assumptions you have to make to be able to go back. Now it should be as easy as going back from sx to ox.

That was the hard part! But you still have to go from the last function to the second one. I guess the second to the first is easy. :) That function is doing a linear matrix transform. Which is good for us. It takes an input of four values (x,y,z) and (w=1) implicit and outputs four other values (ax,ay,az,aw). We could figure out how to go back there manually! I had to do that in school.. four unknowns, four equations. You know ax,ay,az,aw... solve for x,y,z and you get w=1 for free! Very possible and a good exercise but also tedious. The good news is that the way those equations are written is called a matrix. (x,y,z,1) * MODELMATRIX = (ax,ay,az,aw). Really convenient because we can find MODELMATRIX^-1. It's called the inverse! Just like 1/2 is the inverse of 2 for multiplying real numbers or -1 is the inverse of 1 for addition. You really should read up on this it's fun and not hard, btw :). Anyhow, use any standard library to get the inverse of your model matrix. Probably something like modelView.Inverse(). And then do the same function with that and you go backwards. Easy!

Now, why did we not do the same thing with the PROJECTION matrix earlier? Glad you asked! That one takes 4 inputs(x,y,z,w) and spits out only three outputs (screenx,screeny,zbufferz). So without making some assumptions we could not solve it! An intuitive way to look at that is that if you have a 3d point, that you project on a 2d screen, there's going to be a lot of possible solutions. So we have to pick something. And we can not use the convenient matrix inverse function.

Let me know if this was somewhat helpful or not. I have a feeling that it's not, but I had fun writing it! Also google for unproject in processing gives this: http://forum.processing.org/topic/read-3d-positions-gluunproject