What are the main references to the fish-eye camera model in OpenCV3.0.0dev?

Question

I am wrestling with the Fish-Eye Camera Model used in OpenCV 3.0.0.dev. I have read the documentation in this link several times, especially the "Detailed Description" part and formulas modeling fish-eye distortion. By now I have two concerns:

1. Based on the projection models listed here and their conceptual explanations in "Accuracy of Fish-Eye Lens Model" By Hughes, I can't figure out which projection model has been used in the OpenCV implementation.

2. Since the description is so concise, I need to know the main reference papers used by OpenCV developers for implementing fish-eye namespace, so that I could be on the ball and get through more details. P.S. I checked OpenCV 3.0.0-dev documentation and did not find anything useful.

Answer 1

Short answer:

OpenCV 3.0.0 Fisheye camera model doesn't use the Brown model nor any of the models that the OP refers from Panotools, it uses a Generic Camera Model by Juho Kannala and Sami S. Brandt.

Detailed answer:

As Cfr points in his answer, this comment from Ilya Krylov (who implemented the fisheye model in OpenCV) says they ported the Camera Calibration Toolbox for Matlab of Jean-Yves Bouguet:

Jean-Yves Bouguet website (link), in turn, mentions the paper A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses, and says :

The "undocumented" fisheye model contained in the calibration toolbox follows the equidistance projection model described by equation (3) in this very nice paper. The distortion model follows equation (6), to the exception that k1=1 (otherwise indistinguishable from f).

Which in my opinion is a misleading statement or a plain misconception as equation(3) and equation(6) correspond to different models: equation (6) is the actual model introduced in this paper which the authors refer as the Generic Camera Model (hence the name of the paper). To be more precise, equation (6) was meant to be used as the camera model and equation (8) and (9) as the "distortion" or deviation from this model.

But the odyssey is not over. OpenCV's implementation (according to its documentation) first computes the pinhole projection to find the field angle (the angle between the 3D point, the center of projection and the optical axis). This means you can't use their fisheye model to project rays at 90º (or you would divide by 0) or close to 90º (numerical stability problems, like overflow could happen if z is small enough). Moreover I'm not sure whether it will work for rays for more than 90º. All this makes me really wonder the "usefulness" of their fisheye camera model for fisheye or wide angle lenses.

In case you're skeptic about that you can take a look at OpenCV's source code, concretely at sources\modules\calib3d\src\fisheye.cpp (I added some comments)

void cv::fisheye::projectPoints(InputArray objectPoints, OutputArray imagePoints, InputArray _rvec, InputArray _tvec, InputArray _K, InputArray _D, double alpha, OutputArray jacobian) {     ...     Rodrigues(om, R, dRdom);     Affine3d aff(om, T);     ...     Vec3d Xi = objectPoints.depth() == CV_32F ? (Vec3d)Xf[i] : Xd[i];     Vec3d Y = aff*Xi; /* To transform to camera reference frame*/      Vec2d x(Y[0]/Y[2], Y[1]/Y[2]); /* <- The root of all evil (division by z) */      double r2 = x.dot(x);     double r = std::sqrt(r2);      // Angle of the incoming ray:     double theta = atan(r);      double theta2 = theta*theta, theta3 = theta2*theta, theta4 = theta2*theta2, theta5 = theta4*theta,             theta6 = theta3*theta3, theta7 = theta6*theta, theta8 = theta4*theta4, theta9 = theta8*theta;      double theta_d = theta + k[0]*theta3 + k[1]*theta5 + k[2]*theta7 + k[3]*theta9;      double inv_r = r > 1e-8 ? 1.0/r : 1;     double cdist = r > 1e-8 ? theta_d * inv_r : 1;      Vec2d xd1 = x * cdist;     Vec2d xd3(xd1[0] + alpha*xd1[1], xd1[1]);     Vec2d final_point(xd3[0] * f[0] + c[0], xd3[1] * f[1] + c[1]);     ... } 

Update: This pull request fixes the problem with rays at angles ≥ 90º. As of April 2018 it hasn't been merged into master yet but is being considered for OpenCV 4.x Calibration Module (check calibration module discussion too)

Answer 2

After hours of reading, I found that the formula θ=atan(r) in OpenCV's fish-eye documentation, is the normalized inverse of r=f*tanθ pertaining to pinhole projection and hence, none of the fish-eye projection models mentioned in the above links are used in OpenCV.

In addition, regarding the distortion model, what I guess is that the Division Model of Fitzgibbon in his 2001 paper "Simultaneous linear estimation of multiple view geometry and lens distortion" is used. According to Hughes in his 2008 paper "Review of Geometric Distortion Compensation in Fish-Eye Cameras", among the other alternatives are "Odd Polynomial Model" and "Polynomial Fish-Eye Transform". In his paper, in page 2, he has written:

"(1) (which refers to Odd Polynomial Model) and (3) (which refers to Division Model, which I guess is the one used by OpenCV) can be used to describe distortion in standard,non-fisheye lenses. However, it is generally considered that these polynomial models are insufficient to describe the level of distortion introduced by fish-eye lenses. Shah and Aggarwal have shown in [9] (Intrinsic Parameter Calibration Procedure For A High-Distortion Fish-Eye Lens Camera With Distortion Model And Accuracy Estimation) that even when using a 7th order version of (1) to model fish-eye radial distortion, considerable distortion remains, to the extent that they had to use a model with greater degrees of freedom. Therefore, a polynomial that uses both odd and even coefficients (instead of simply one or the other) can be used to model the radial distortion introduced by a fisheye lens"

After all, I conclude that the fish-eye model in OpenCV has very limited applicability and could be much more strengthened in terms of distortion models and projection models. I'd like to re-emphasize that I still need to know what papers were used by the OpenCV developers to implement the fisheye namespace.

I'd deeply appreciate anyone's comments on this.

Answer 3

according to Bouguet's calib_tool code "project_points_fisheye.m"

%Definitions:
%Let P be a point in 3D of coordinates X in the world reference frame    (stored in the matrix X)
%The coordinate vector of P in the camera reference frame is: Xc = R*X + T
%where R is the rotation matrix corresponding to the rotation vector om: R = rodrigues(om);
%call x, y and z the 3 coordinates of Xc: x = Xc(1); y = Xc(2); z = Xc(3);
%The pinehole projection coordinates of P is [a;b] where a=x/z and b=y/z.
%call r^2 = a^2 + b^2,
%call theta = atan(r),
%Fisheye distortion -> theta_d = theta * (1 + k(1)*theta^2 + k(2)*theta^4 + k(3)*theta^6 + k(4)*theta^8)
%
%The distorted point coordinates are: xd = [xx;yy] where:
%
%xx = (theta_d / r) * x
%yy = (theta_d / r) * y
%
%Finally, convertion into pixel coordinates: The final pixel coordinates    vector xp=[xxp;yyp] where:
%
%xxp = f(1)*(xx + alpha*yy) + c(1)
%yyp = f(2)*yy + c(2)