What is an "acceptable" return value from cv::calibrateCamera?

Question

Background:

I'm currently working on a stereo vision project using OpenCV. I'm trying to create a disparity map from a set of two rectified images, but I'm not getting the result I expect. When looking at the rectified images, there is a noticeable vertical shift between the images, which should not be there after rectification. I'm currently looking for what the error may be. My code is based on the stereo calibration and correspondence code from the OpenCV book, and this example. I use the C++ interface of OpenCV. My OpenCV version is 2.1, from the Ubuntu 11.04 repository.

Short version of question:

What RMS return value is acceptable for the function:

double cv::calibrateCamera(...)

I currently calibrate each camera using a set of ~20 chessboard image pairs. The cameras are two identical PS3 Eyes (with a resolution of 640*480 pixels) mounted beside each other. cv::calibrateCamera returns an RMS error of between 160 and 300 (I've had different result with different image sets). Is this an acceptable value for this image resolution or should I attempt to get better chessboard images?

Long version (details, code samples):

To get working stereo vision; first, I want to be sure the camera calibration routine works correctly. I use a series of chessboard images to calibrate my stereo setup, like so:

// Find chessboard corners
cv::Mat left = ... //Load image
vector<cv::Point2f> points;

bool found = cv::findChessboardCorners(left, patternSize, points, CV_CALIB_CB_ADAPTIVE_THRESH | CV_CALIB_CB_NORMALIZE_IMAGE);
if(found)
  cv::cornerSubPix(left, points, cv::Size(11, 11), cv::Size(-1, -1), cv::TermCriteria(CV_TERMCRIT_EPS + CV_TERMCRIT_ITER, 30, 0.1));
imagePointsLeft.push_back(points);

I do this for a number of image pairs and put the resulting points in imagePointsLeft and imagePointsRight. I have run a check in a separate program on the same images where I visually confirm that the left and right images correctly register all corners, in the same order, using cv::drawChessboardCorners. For each image pair I also fill a set of object coordinates like so:

vector<cv::Point3f> obj;
for(int i = 0; i < ny; i++)
  for(int j = 0; j < nx; j++)
    obj.push_back(cv::Point3f(i*squareSize,j*squareSize,0.0));
objectPoints.push_back(obj);

I then attempt to calibrate the cameras with these images:

double rms = cv::calibrateCamera(objectPoints, imagePointsLeft, m_imageSize, m_cameraMatrixLeft, m_distortionMatrixLeft, rvecsLeft, tvecsLeft);

I use the camera intrinsics from this function when I call cv::stereoCalibrate using the flag CV_CALIB_FIX_INTRINSIC (this way of stereo calibration is suggested in the documentation, here).

I proceed to call cv::stereoRectify and cv::initUndistortRectifyMap, the latter for both cameras. The rectified images are produced with cv::remap using the output of initUndistortRectifyMap and a camera livefeed. Both images contain only valid pixels (no black areas). My problem is that the pixels of a given row does not match the same row in the other image (there is a slight vertical offset). This makes it difficult to get any good results from correspondence algorithms such as StereoBM.

Answer 1

There is a problem with your camera calibration: cv::calibrateCamera() returns the root mean square (RMS) reprojection error [1] and should be between 0.1 and 1.0 pixels in a good calibration. For a point of reference, I get approximately 0.25 px RMS error using my custom stereo camera made of two hardware-synchronized Playstation Eye cameras running at the 640 x 480 resolution.

Are you sure that the pixel coordinates returned by cv::findChessboardCorners() are in the same order as those in obj? If the axes were flipped, you would get symptoms similar to those that you are describing.

[1]: OpenCV calculates reprojection error by projecting three-dimensional of chessboard points into the image using the final set of calibration parameters and comparing the position of the corners. An RMS error of 300 means that, on average, each of these projected points is 300 px away from its actual position.