OpenCV - Calibrate fisheye lens error (Ill-conditioned matrix)

Question

I am trying to calibrate a fisheye lens following these instructions https://medium.com/@kennethjiang/calibrate-fisheye-lens-using-opencv-333b05afa0b0 where you can find the full code I'm using for the calibration part.

I arrive at this point where:

N_OK = len(objpoints)
K = np.zeros((3, 3))
D = np.zeros((4, 1))
rvecs = [np.zeros((1, 1, 3), dtype=np.float64) for i in range(N_OK)]
tvecs = [np.zeros((1, 1, 3), dtype=np.float64) for i in range(N_OK)]  
rms, _, _, _, _ = \
    cv2.fisheye.calibrate(
        objpoints,
        imgpoints,
        gray.shape[::-1],
        K,
        D,
        rvecs,
        tvecs,
        calibration_flags,
        (cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3)
    )
print("Found " + str(N_OK) + " valid images for calibration")
print("DIM=" + str(_img_shape[::-1]))
print("K=np.array(" + str(K.tolist()) + ")")
print("D=np.array(" + str(D.tolist()) + ")")

I get this error:

Traceback (most recent call last)
<ipython-input-10-deaca9981fe4> in <module>()
     13         tvecs,
     14         calibration_flags,
---> 15         (cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3)
     16     )
     17 print("Found " + str(N_OK) + " valid images for calibration")

error: C:\ci\opencv_1512688052760\work\modules\calib3d\src\fisheye.cpp:1414: 
error: (-3) CALIB_CHECK_COND - Ill-conditioned matrix for input array 0 in 
function cv::internal::CalibrateExtrinsics

I don't understand what's going on and I could only find so little information around the internet, does anyone have experienced something similar and know how to solve this?

Thanks

These are the images of the checkerboard I'm using:

• https://i.stack.imgur.com/goHIg.jpg
• https://i.stack.imgur.com/kja3O.jpg
• https://i.stack.imgur.com/XbIzh.jpg
• https://i.stack.imgur.com/uQ9Gr.jpg
• https://i.stack.imgur.com/N8aLJ.jpg
• https://i.stack.imgur.com/JMmmZ.jpg
• https://i.stack.imgur.com/QJ8wn.jpg
• https://i.stack.imgur.com/NmiFQ.jpg
• https://i.stack.imgur.com/sQHdd.jpg
• https://i.stack.imgur.com/DNBuD.jpg
• https://i.stack.imgur.com/vFtOo.jpg
• https://i.stack.imgur.com/lhkF5.jpg

Answer 1

I think it is because your variable calibration_flags has CALIB_CHECK_COND set. Try disabling this flag. Without it I was able to undistort your images (see links below).

I am not sure what this check is for (the documentation is not very explicit). This flag reject some images¹ of my gopro hero 3 even when the chessboard is visible and detected. In my case one image among 20 is not passing this test. This image has the chessboard close to the left border.

• https://i.stack.imgur.com/m2WF6.jpg
• https://i.stack.imgur.com/KiTRz.jpg
• https://i.stack.imgur.com/MhHyN.jpg
• https://i.stack.imgur.com/pSiyG.jpg
• https://i.stack.imgur.com/drXSL.jpg
• https://i.stack.imgur.com/DDze1.jpg
• https://i.stack.imgur.com/b6l8f.jpg
• https://i.stack.imgur.com/9MrAk.jpg
• https://i.stack.imgur.com/WYmg5.jpg
• https://i.stack.imgur.com/rmJ5Q.jpg
• https://i.stack.imgur.com/K8k8Y.jpg
• https://i.stack.imgur.com/MiBER.jpg

¹ in OpenCV versions >= 3.4.1 the error message tells you which image is not passing the test

Answer 2

I did not find the code in python so I manually check the images with chessboard on the edge and delete them one by one until the error is gone.

Answer 3

As @Ahmadiah mentioned, the "ill conditioned" thing can happen when the checkerboard falls near the edge of the image. One way to handle this is to remove images one by one and try again when they cause calibration to fail. Here is an example where we do that:

def calibrate_fisheye(all_image_points, all_true_points, image_size):
    """ Calibrate a fisheye camera from matching points.
    :param all_image_points: Sequence[Array(N, 2)[float32]] of (x, y) image coordinates of the points.  (see  cv2.findChessboardCorners)
    :param all_true_points: Sequence[Array(N, 3)[float32]] of (x,y,z) points.  (If from a grid, just put (x,y) on a regular grid and z=0)
        Note that each of these sets of points can be in its own reference frame,
    :param image_size: The (size_y, size_x) of the image.
    :return: (rms, mtx, dist, rvecs, tvecs) where
        rms: float - The root-mean-squared error
        mtx: array[3x3] A 3x3 camera intrinsics matrix
        dst: array[4x1] A (4x1) array of distortion coefficients
        rvecs: Sequence[array[N,3,1]] of estimated rotation vectors for each set of true points
        tvecs: Sequence[array[N,3,1]] of estimated translation vectors for each set of true points
    """
    assert len(all_true_points) == len(all_image_points)
    all_true_points = list(all_true_points)  # Because we'll modify it in place
    all_image_points = list(all_image_points)
    while True:
        assert len(all_true_points) > 0, "There are no valid images from which to calibrate."
        try:
            rms, mtx, dist, rvecs, tvecs = cv2.fisheye.calibrate(
                objectPoints=[p[None, :, :] for p in all_true_points],
                imagePoints=[p[:, None, :] for p in all_image_points],
                image_size=image_size,
                K=np.zeros((3, 3)),
                D=np.zeros((4, 1)),
                flags=cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC + cv2.fisheye.CALIB_CHECK_COND + cv2.fisheye.CALIB_FIX_SKEW,
                criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-6),
            )
            print('Found a calibration based on {} well-conditioned images.'.format(len(all_true_points)))
            return rms, mtx, dist, rvecs, tvecs
        except cv2.error as err:
            try:
                idx = int(err.message.split('array ')[1][0])  # Parse index of invalid image from error message
                all_true_points.pop(idx)
                all_image_points.pop(idx)
                print("Removed ill-conditioned image {} from the data.  Trying again...".format(idx))
            except IndexError:
                raise err