How do you triangulate a 3D point from a pair of stereo images?

Question

I'm working with a stereo pair of photos, and trying to obtain a set of 3D points from matching points on these 2 images. How exactly is the 3D point triangulated? Do you throw a ray from the camera pinhole centerpoint? Or do you throw a ray perpendicular to the photos?

Answer 1

You should throw the ray from the camera center point.

Using parallel rays would only be appropriate if the original images were created with an orthographic projection.

Answer 2

The reconstruction of 3D points is significantly simplified if your images are rectified. This means that cameras are effectively parallel (looking at the same direction) and even if they are not the images can be transformed to satisfy this requirement. Then all you need to do to restore 3D is to know cameras effective horizontal displacement (baseline). In short, Z = fB/D, X = uZ/f, Y = vZ/f, where B is a baseline, D - disparity, f-focal length and u, v is column and -row with the origin in the center of the image.

However, what you showed in the picture is not a pair of cameras with parallel optical axis. If you don’t want to rectify their images you have to take into account rotation and translation between cameras, which will complicate reconstruction.