Single Camera Calibration with OpenCV - Problems generating a "Complete" undistorted image

Question

I've been attempting to undistort imagery from a fisheye camera (if it's relevant, I'm using a GoPro) using OpenCV's camera calibration suite. I've got most of the process working and can generate undistorted images. However, when using remap, the undistorted image is the "valid rectangle" - in other words, the image returned is a cropped version of the original to avoid the curved black borders that are inherent in undistorted frames.

I have attempted to use getOptimalNewCameraMatrix() to correct the situation, with very strange results. I'm hoping one of you can shed some light on my problems.

I currently calibrate the camera as follows:

double error = calibrateCamera(worldPoints, sensorPoints, process_size, cameraMatrix, distCoeffs, rvecs, tvecs, calibration_flags);

which generates the cameraMatrix I need. This part works (I think), because if I run this through initUndistortRectifyMap() followed by remap(), I get a valid image back. However, I'm looking for the full image, so the next thing I do is try to correct my cameraMatrix as follows:

int alpha = 1;
cameraMatrix_corr = getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, image.size(), alpha);

Then, I generate the X and Y maps for remap, as follows (this is borrowed directly from the OpenCV 2.0 Cookbook, so I'm fairly confident it works too).

initUndistortRectifyMap(
    cameraMatrix_corr,  // computed camera matrix
    distCoeffs, // computed distortion matrix
    Mat(), // optional rectification (none)
    Mat(), // camera matrix to generate undistorted
    image.size(),  // size of undistorted
    CV_32FC1,      // type of output map
    map1, map2);   // the x and y mapping functions

Finally, I remap my image!

// Apply mapping functions
remap(image, undistorted, map1, map2, INTER_LINEAR);

Here are my results so far. If I use alpha = 0 (no change to the matrix), I get reasonable (though cropped) results.

If I use alpha = 1, which I think should give me an image with every original pixel mapped to the new image, I get the following:

So, my question is this: What am I doing wrong, and why can I not just get an un-cropped, undistorted image out of the calibration?

Thanks all for putting up with me, this is my first question here, but I tried to be as complete as possible. Let me know if I screwed up somehow!

Answer 1

I believe correcting the camera matrix is not something you should do for undistorting, because that would give you a camera matrix as if there is no distortion.

If you don't want to "lose" image information it is probably best to shift the map in x and y and map the image to a bigger image. This is because when undistorting the pixels in the corners will move outward to rectify the image.

Do keep in mind that fisheye lenses have high distortions and might not even be fully rectified even when using the correct code.