Open CV - Several Methods for SfM

Question

I got a task:

We have a system working where a camera does a halfcircle around a human head. We know the camera matrix and the rotation/translation of every frame. (Distortion and more... but I want first to work without these parameters)

My task is that I have only the Camera Matrix, which is constant over this move, and the images (more than 100). Now I have to get the translation and rotation from frame by frame and compare it with the rotation and translation in real world (from the system which I have but only for compare, I have too prove it!)

First steps I did so far:

1. use the robustMatcher from the OpenCV Cookbook - works finde - 40-70 Matches each frame - visible looks it very good!
2. I get the fundamentalMatrix with getFundamental(). I use the robust Points from robustMatcher and RANSAC.
3. When I got the F i can get the Essentialmatrix E with my CameraMatrix K like this:

cv::Mat E = K.t() * F * K; //Found at the Bible HZ Chapter 9.12

Now we need to extract R and t out of E with SVD. By the way camera1 position is just zero because we have to start somewhere.

cv::SVD svd(E);
cv::SVD svd(E);

cv::Matx33d W(0,-1,0,   //HZ 9.13
          1,0,0,
          0,0,1);

 cv::Matx33d Wt(0,1,0,//W^
        -1,0,0,
        0,0,1);

 cv::Mat R1 = svd.u * cv::Mat(W)  * svd.vt; //HZ 9.19
 cv::Mat R2 = svd.u * cv::Mat(Wt) * svd.vt; //HZ 9.19

 //R1 or R2???
 R = R1; //R2

 //t=+u3 or t=-u3?
 t = svd.u.col(2); //=u3

This is my actual status!

My plans are:

1. triangulate all points to get 3D points
2. Join frame i with frame i++
3. Visualize my 3D points them somehow!

Now my Questions are:

1. is this robust matcher dated? is there a other method?
2. Is it wrong to use this points as descriped at my second step? Must they be converted with distortion or something?
3. What R and t is this i extract here? Is it the rotation and translation between camera1 and camera2 with point of view from camera1?
4. When I read at the bible or papers or elsewhere i find that there are 4 possibilities how R and t can be! ´P′ = [UWV^T |+u3] oder [UWV^T |−u3] oder [UW^TV^T |+u3] oder [UW^TV^T |−u3]´ P´ is the projectionmatrix of the second image. That means t could be - or + and R could be total different?! I found out that I should calculate one point into 3D and find out if this point is infront of both cameras, then I have found the correct matrix! I found some of this code at the internet and he just said this no further calculating: cv::Mat R1 = svd.u * cv::Mat(W) * svd.vt and t = svd.u.col(2); //=u3 Why is this correct? If it isn't - how would I do this triangulation in OpenCV? I compared this translation to the translation which is given to me. (First i had to transfer the translation and rotation in relationship to camera1 but I got this now!) But its not the same. The values of my program are just lets call it jumping from plus too minus. But it should be more constant because the camera is moving in a constant circle. I am sure that some axes may be switched. I know that the translation is only from -1 till 1 but I thought I could extract a factor from my results to my comparevalues and then it should be similiar.

Does somebody have done something like this before?

Many people doing a camera calibration by using a chessboard, but I can't use this method to get the extrinsic parameters.

I know that visual sfm can do this somehow. (At youtube is a video where someone walks around a tree and get from these pictures a reconstruction of this tree using visual sfm) This is pretty the same what I have to do.

Last question:

Does somebody know an easy way to visualize my 3D Points? I prefere MeshLab. Some experience with that?

Answer 1

Many people doing a camera calibration by using a chessboard, but I can't use this method to get the extrinsic parameters.

A chess board or checker board is used to find the internal/intrinsic matrix/parameters, not the extrinsic parameters. You're saying you have got the internal matrix already, I suppose that's what you meant by

We know the camera matrix and ...

Those videos you have seen on youtube have done the same, the camera is already calibrated, that is the internal matrix is known.

is this robust matcher dated? is there a other method?

I don't have that book so cant see the code and answer this.

Is it wrong to use this points as descriped at my second step? Must they be converted with distortion or something?

You need to cancel the radial distortion first, see undistortPoints.

What R and t is this i extract here? Is it the rotation and translation between camera1 and camera2 with point of view from camera1?

R is the orientation of the second camera in the first camera's coordinate system. And T is position of the second camera in that coordinate system. These have several usages.

When I read at the bible or papers or elsewhere i find that there are 4 possibilities how ....

Read the relevant section of the bible, this is very well explained there, triangulation is naive method, a better approach is explained there.

Does somebody know an easy way to visualize my 3D Points?

To see them in Meshlab a very easy way is to save the coordinate of the 3D points in a PLY file, this is an extremely simple format and supported by Meshlab and almost all other 3D model viewers.