How to use opencv bundle adjustment

Question

I have calculated 5 camera matrices (c1, ... c5), camera matrices are calculated by placing a 3D object at 5 different positions and for each position I have calculated camera matrix (and camera is constant). Camera matrix is calculated using SVD method.

Now I want to use bundle adjustment in opencv to get one optimal camera matrix. I found the documentation here

But documentation is not clear and also I was not able to find any sample code. Can anybody explain how can I use opencv bundle adjustment to get the optimal camera matrix ?

Answer 1

My answer assumes that you use a specific 3D object, whose dimensions are known accurately, to estimate the intrinsic parameters of one camera from multiple images of the object. In light of your comment to @fireant 's answer, I think this applies to your question.

Quick note on the meanings of 'camera matrix'

The term 'camera matrix' is very vague and can be understood in multiple ways:

• The camera pose T = [R|t], also denoted extrinsic camera parameters.
• The intrinsic camera matrix K = [fx, s, cx; 0, fy, cy; 0, 0, 1].
• The camera projection matrix P = K.[R|t].

As you say that you have multiple images but you want a single camera matrix, I think that you are talking about the intrinsic camera matrix K.

Why you should not use Bundle Adjustment

Bundle Adjustment is a technique used to solve a more general problem than camera calibration, where both the extrinsic camera parameters (i.e. 3D orientations and positions) and the 3D landmarks (e.g. 3D points) are unknown. As @fireant noted, it is also possible to include intrinsic camera parameters in this global joint optimization. On the other hand, camera calibration assumes the 3D landmarks are known and observed by a single camera in multiple images, therefore it optimizes a single set of intrinsic camera parameters and one camera pose for each image.

The thing to understand is that more general optimization problems involve more variables to be optimized, and it is therefore harder to make them well-constrained. If they are not well-constrained, they will optimize your variables in a way that indeed decrease the global error, but that does not correspond to a solution of your real problem. This is why, when using a joint optimization algorithm, you should always try to reduce the number of variables to be optimized.

In the case of Bundle Adjustment VS Camera Calibration, you should only use Bundle Adjustment if you have no accurate idea of positions of the 3D landmarks. Even then, it is probably a better idea to try and decouple the problems, for instance by calibrating the 3D object beforehand. If you do have an accurate idea of the positions of the 3D landmarks, then they should not be considered as variables to be optimized, hence you should use the Camera Calibration technique.

Initial estimate for the optimization

You are saying that you have multiple inaccurate estimates of the camera matrix K, and want to perform a joint optimization to obtain a single more accurate one. The problem is that, since you want to estimate a single camera matrix, you need to provide the joint optimization algorithm with only one initial estimate.

What you could do, in order to still use your multiple approximate estimates, is to try to select, as initial estimate for the optimization algorithm, the one closest to the true solution. For this purpose, you can use several heuristic criteria. For example you can select the one associated with the minimum reprojection error, or you can use the one associated to the image where the calibration object is the largest, etc.

However, it will probably not make a big difference in the final estimate of the camera matrix.

How to use 'calibrateCamera' for this task

Suppose you have a calibration object with characteristic points. It can be a standard 2D chessboard or assymetric circle grid, or any calibrated 3D object. And suppose you have developped a way to detect this calibration object in an image (e.g. a custom feature detector, or a tool to manually determine the position of the object). Then you can define a vector allObjectPoints, containing the 3D points on the object that can be detected. Then for each image i, you can determine a vector imagePoints_i of the 2D points detected to be observations of some the object's 3D points (note that some object points may be occluded, hence there may be less items than in allObjectPoints). Then, since all the object points are identifiable (either directly or by reasoning), you can determine the vector objectPoints_i, containing the object 3D points actually observed in image i and ordered consistantly with imagePoints_i.

Then you stack all the objectPoints_i vectors in one big vector of vectors objectPoints. Similarly you stack all the imagePoints_i vectors in one big vector of vectors imagePoints. You can then make the call to calibrateCamera, with the flag CV_CALIB_USE_INTRINSIC_GUESS to indicate that you want the optimization algorithm to use the initial value that you provided.

Answer 2

See the sample code at page 155 of Learning Image Processing with OpenCV. Something like this:

vector<CameraParams> cameras;
vector<MatchesInfo> pairwise_matches;
vector<ImageFeatures> features(num_images);

// initialize the above params here

Ptr<BundleAdjusterBase> adjuster;
adjuster = makePtr<BundleAdjusterReproj>();
if (!(*adjuster)(features, pairwise_matches, cameras)) {
    cout << "Camera parameters adjusting failed." << endl; 
    return -1;
}

If you provide a MCVE then it's easier to help.

Edit: Given you have 5 estimates of the same K matrix. The easiest method is to simply average your 5 estimations to get a more accurate K. Under some mild assumptions that will be an optimal estimation. If the reprojection errors vastly differ, then you can calculate a weighted average.