OpenCV fisheye calibration cuts too much of the resulting image

Question

I am using OpenCV to calibrate images taken using cameras with fish-eye lenses.

The functions I am using are:

• findChessboardCorners(...); to find the corners of the calibration pattern.
• cornerSubPix(...); to refine the found corners.
• fisheye::calibrate(...); to calibrate the camera matrix and the distortion coefficients.
• fisheye::undistortImage(...); to undistort the images using the camera info obtained from calibration.

While the resulting image does appear to look good (straight lines and so on), my issue is that the function cut away too much of the image.

This is a real problem, as I am using four cameras with 90 degrees between them, and when so much of the sides are cut off, there is no overlapping area between them which is needed as I am going to stitch the images.

I looked into using fisheye::estimateNewCameraMatrixForUndistortRectify(...) but I could not get it to give good results, as I do not know what I should put in as the R input, as the rotation vector output of fisheye::calibrate is 3xN (where N is the number of calibration images) and fisheye::estimateNewCameraMatrixForUndistortRectify requires a 1x3 or 3x3.

The images below show an image of my undistortion result, and an example of the kind of result I would ideally want.

Undistortion:

Example of wanted result:

Answer 1

I think I have ran into a similar issue, looking for the "alpha" knot in getOptimalNewCameraMatrix for fisheye.

Original shot:

I calibrated it with cv2.fisheye.calibrate, got the K and D parameters

K = [[ 329.75951163    0.          422.36510555]
 [   0.          329.84897388  266.45855056]
 [   0.            0.            1.        ]]

D = [[ 0.04004325]
 [ 0.00112638]
 [ 0.01004722]
 [-0.00593285]]

This is what I get with

map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, d, np.eye(3), k, (800,600), cv2.CV_16SC2)
nemImg = cv2.remap( img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

And I think it chops too much. I want to see the whole Rubik's cube

I fix it with

nk = k.copy()
nk[0,0]=k[0,0]/2
nk[1,1]=k[1,1]/2
# Just by scaling the matrix coefficients!

map1, map2 = cv2.fisheye.initUndistortRectifyMap(k, d, np.eye(3), nk, (800,600), cv2.CV_16SC2)  # Pass k in 1st parameter, nk in 4th parameter
nemImg = cv2.remap( img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

TADA!