Finding Rotation matrices between two cameras for "Stereorectify"

Question

So I have a depth map and the extrinsics and intrinsics of the camera.I want to get back the 3D points and the surface normals .I am using the functionReprojectImageTo3D.In the stereo rectify function to find Q how do I get the The rotation matrix between the 1st and the 2nd cameras’ coordinate systems? I have individual rotation matrix and translation vector but how do I get it for "between the cameras?" .Also this would give me the 3D points .Is there a method to generate the surface normals?

Answer 1

Given that you have the extrinsic matrix of both cameras, can't you simply take the inverse extrinsic matrix of camera 1, multiplied by the extrinsic matrix of camera 2? Also, for a direct relation between the two cameras, take a look at the Fundamental Matrix (or, more specific, the Essential matrix). See if you can find a copy of the book Multiple View Geometry by Hartley and Zisserman.

As for the surface normals, you can compute those yourself by computing crossproducts on the corners of triangles. However, you then first need the reconstructed 3D point cloud.