Having three points on 3 images from 3 viewpoints how to get its coordinates in 3d space?

Question

I have 3 viewpoints locations (x,y,z coordinates on 3d grid), directions (relative to viewpoints origin points x,y,z vector) in 3d space. They all look and each does see exactly three points (markers (x,y) with filtered out background) in space (say we have a red, blue, green dot on white images). We do not know any other viewpoints-camera properties except resolution and that this are same (extremely similar) cameras. How could we get our point position in space?

Answer 1

I assume you have your 3 cameras calibrated, otherwise I do not know if a solution exists to your problem.

If you have your cameras calibration parameters i.e. camera matrix (A), rotation vector (T), translation vector (R). From these you can get the projection matrix using P = A[R|T]. This means that you can project any 3d point in the real world X to image i using X.Pi = x which means you can get a system of linear equations for each view using xi' = xi.P3T - P1T and yi' = yi.P3T - P2T PiT is the transpose of the ith column in P. You can solve this system of equations using singular value decomposition.

References:

1. http://www.robots.ox.ac.uk/~az/tutorials/tutoriala.pdf
2. https://github.com/opencv/opencv_contrib/blob/master/modules/sfm/src/triangulation.cpp
3. http://www.morethantechnical.com/2012/01/04/simple-triangulation-with-opencv-from-harley-zisserman-w-code/

Answer 2

You need one more piece of information: your camera's field of view:

https://blog.codinghorror.com/content/images/uploads/2007/08/6a0120a85dcdae970b0120a86d9495970b-pi.png

I'll refer to horizontal field of view as hFOV and vertical field of view as vFOV. You only need one of these numbers to solve your problem though, as hFOV/vFOV is equal to the aspect ratio (horizontal resolution / vertical resolution).

With that information, you can use the camera's position (Pc, a 3D vector), direction (Dc, a 3D vector), resolution (R, a 2D vector) and the point's position in view space (Pp, a 2D vector) to build a 3D vector that the point lies upon in space.

If the point lies exactly in the middle of the view (Pp / R = 0.5) then this vector is simply Pc + d * Dc, where d is any positive number, corresponding to the distance from the camera (this will become our variable that we need to solve for.

The x coordinate of Pp rotates Dc in x from -hFOV/2 to hFOV/2 about the vector perpendicular to both Dc and the x direction of the screen (you can find this with the cross product). The y coordinate does the same in y. We'll call this rotated vector D'c. Rotating a vector by θ about another vector is solved here: Rotating a Vector in 3D Space

So now we have the relationship:

Position of marker = Pc + d * D'c

Now, d is still an unknown (distance from camera). But if we do this for two cameras, we can solve the system of equations and find the absolute position of the marker. The third camera is useful for verification.

Note: This is more or less the same answer as @osama-alkoky, just the manual version. Learning about projection/transformation matrices is a good idea if you're going to be doing a lot of this.