The coordinate system of pinhole camera model

Question

Recently, I have been studying the pinhole camera model, but I was confused with the model provided by OpenCV and the "Multiple View geometry in computer vision" textbook.

I know that the following photo is a simplified model which switch the position of the image plane and the camera frame. For better illustration and understanding, and taking consideration of the principal point (u0,v0), the relation between two frames is x=f(X/Z)+u0 and y=f(Y/Z)+v0.

However,I was really confused because normally the image coordinate is in the form of the 4th quadrant coordinate as the following one!

Could I directly substitute the (x,y) in the following definition to the above "equivalent" pinhole model which is not really persuasive?

Besides, If an object is in the region (+X,+Y) quadrant in the camera coordinate (of course, Z>f), in the equivalent model, it should appear on the right-half plane of the image coordinate. However, such object in the image taken by a normal camera, it is supposed to be located on the left-half. Therefore, for me this model is not reasonable.

Finally, I tried to derive based on the original model as the following one.

The result is x1=-f(X/Z) and y1=-f(Y/Z).

Then, I tried to find the relation between (x2,y2)-coordinate and the camera coordinate. The result is x2=-f(X/Z)+u0 and y2=-f(Y/Z)+v0.

Between (x3,y3)-coordinate and the camera coordinate, the result is x3=-f(X/Z)+u0 and y3=f(Y/Z)+v0.

No matter which coordinate system I tried, none of them is in the form of x=f(X/Z)+u0 and y=f(Y/Z)+v0, which are provided by some CV textbooks.

Besides, the projection results on (x2,y2)-coordinate or (x3,y3)-coordinate are also not reasonable because of the same reason: an object in the (+X,+Y,+Z) region in the camera coordinate should "appear" on the left-half plane of the image taken by a camera.

Could anyone indicate what I misunderstood?

Answer 1

I finally figured out this issue and proved my interpretation is right by implementing the paper by Z. Zhang, Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999.

Let me explain everything from scratch. The following photo is the original pinhole camera model and the projected result on the image sensor. However, this is not what we are supposed to see on the "image".

What we should see is

Comparing figure 1 and 2, we should notice that this picture is up-side-down and left-side-right. My friend who works for a CMOS sensor company told me that there are built-in functions to automatically flip the perceived image.

Since we want to model the relationship between image coordinate and the world coordinate, we should directly treat the image sensor as a projection plane. What confused me previously is the projection is always limited to the projected side and this misled me to geometrically understand the derivation.

Now, we should look from the "back" of the image sensor as the blue (View Perspective) arrow.

The result is as figure 2. the x1-y1 coordinate is now toward right and down respectively, so the equations are

x1=-f(X/Z)
y1=-f(Y/Z)

Now, in terms of the x-y coordinate, the equation is

x=f(X/Z)+u0
y=f(Y/Z)+v0

which are what the paper described.

Now, let us take a look at the equivalent model which doesn't exist in real world but helps visual interpretation.

The principle is the same. Look from the center of projection and towards the image plane. The result is

where the projected "F" is right-side-left. The equations are

x1=f(X/Z)
y1=f(Y/Z)

Now, in terms of the x-y coordinate, the equation is

x=f(X/Z)+u0
y=f(Y/Z)+v0

which are what the paper described.

Last but not least, since the unit in world coordinate is mm or inch and the one in image coordinate is pixels, there is a scaling factor where some books describe as

x=a*f(X/Z)+u0 
y=b*f(Y/Z)+v0

or

x=fx(X/Z)+u0
y=fy(Y/Z)+v0

where fx=a*f, fy=b*f