Minimum number of chessboard images for Stereo Calibration and Rectification

Question

What is the minimum number of chessboard image pairs in order to mathematically calibrate and rectify two cameras ? One pair is considered as a single view of the chessboard by each camera, ending with a left and right image of the same scene. As far as I know we need just one pair for a stereo system, as the stereo calibration seeks the relations between the tow cameras.

Answer 1

Stereo calibration seeks not only the rotation and translation between the two cameras, but also the intrinsic and distortion parameters of each camera. You need at least two images to calibrate each camera separately, just to get the intrinsics. If you have already calibrated each camera separately, then, yes, you can use a single pair of checkerboard images to get R and t. However, you will not get a very good accuracy.

As a rule of thumb, you need 10-20 image pairs. You need enough images to cover the field of view, and to have a good distribution of 3D orientations of the board.

To calibrate a stereo pair of cameras, you first calibrate the two cameras separately, and then you do another joint optimization of the parameters of both cameras plus the rotation and translation between them. So one pair of images will simply not work.

Edit: The camera calibration algorithm used in OpenCV, Caltech Calibration Toolbox, and the Computer Vision System Toolbox for MATLAB is based on the work by Zhengyou Zhang. His paper explains it better than I ever could.

The crux of the issue here is that the points on the chessboard are co-planar, which is a degenerate configuration. You simply cannot solve for the intrinsics using just one view of a planar board. You need more than one view, with the board in different 3-D orientations. Views where the boards are in parallel planes do not add any information.