OpenCV 2.3 camera calibration

Question

I'm trying to use OpenCV 2.3 python bindings to calibrate a camera. I've used the data below in matlab and the calibration worked, but I can't seem to get it to work in OpenCV. The camera matrix I setup as an initial guess is very close to the answer calculated from the matlab toolbox.

import cv2
import numpy as np

obj_points = [[-9.7,3.0,4.5],[-11.1,0.5,3.1],[-8.5,0.9,2.4],[-5.8,4.4,2.7],[-4.8,1.5,0.2],[-6.7,-1.6,-0.4],[-8.7,-3.3,-0.6],[-4.3,-1.2,-2.4],[-12.4,-2.3,0.9],    [-14.1,-3.8,-0.6],[-18.9,2.9,2.9],[-14.6,2.3,4.6],[-16.0,0.8,3.0],[-18.9,-0.1,0.3],    [-16.3,-1.7,0.5],[-18.6,-2.7,-2.2]]
img_points = [[993.0,623.0],[942.0,705.0],[1023.0,720.0],[1116.0,645.0],[1136.0,764.0],[1071.0,847.0],[1003.0,885.0],[1142.0,887.0],[886.0,816.0],[827.0,883.0],[710.0,636.0],[837.0,621.0],[789.0,688.0],[699.0,759.0],[768.0,800.0],[697.0,873.0]]

obj_points = np.array(obj_points)
img_points = np.array(img_points)

w = 1680
h = 1050
size = (w,h)

camera_matrix = np.zeros((3, 3))
camera_matrix[0,0]= 2200.0
camera_matrix[1,1]= 2200.0
camera_matrix[2,2]=1.0
camera_matrix[2,0]=750.0
camera_matrix[2,1]=750.0 

dist_coefs = np.zeros(4)
results = cv2.calibrateCamera(obj_points, img_points,size, 
    camera_matrix, dist_coefs)

Answer 1

First off, your camera matrix is wrong. If you read the documentation, it should look like:

fx  0 cx
 0 fy cy
 0  0  1

If you look at yours, you've got it the wrong way round:

fx  0  0
 0 fy  0
cx cy  1

So first, set camera_matrix to camera_matrix.T (or change how you construct camera_matrix. Remember that camera_matrix[i,j] is row i, column j).

camera_matrix = camera_matrix.T

Next, I ran your code and I see that "can't seem to get it to work" means the following error (by the way - always say what you mean by "can't seem to get it to work" in your questions - if it's an error, post the error. If it runs but gives you weirdo numbers, say so):

OpenCV Error: Assertion failed (ni >= 0) in collectCalibrationData, file /home/cha66i/Downloads/OpenCV-2.3.1/modules/calib3d/src/calibration.cpp, line 3161
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: /home/cha66i/Downloads/OpenCV-2.3.1/modules/calib3d/src/calibration.cpp:3161: error: (-215) ni >= 0 in function collectCalibrationData

I then read the documentation (very useful by the way) and noticed that obj_points and img_points have to be vectors of vectors, because it is possible to feed in sets of object/image points for multiple images of the same chessboard(/calibration points).

Hence:

cv2.calibrateCamera([obj_points], [img_points],size, camera_matrix, dist_coefs)

What? I still get the same error?!

Then, I had a look at the OpenCV python2 samples (in the folder OpenCV-2.x.x/samples/python2), and noticed a calibration.py showing me how to use the calibration functions (never underestimate the samples, they're often better than the documentation!).

I tried to run calibration.py but it doesn't run because it doesn't supply the camera_matrix and distCoeffs arguments, which are necessary. So I modified it to feed in a dummy camera_matrix and distCoeffs, and hey, it works!

The only difference I can see between my obj_points/img_points and theirs, is that theirs has dtype=float32, while mine doesn't.

So, I change my obj_points and img_points to also have dtype float32 (the python2 interface to OpenCV is funny like that; often functions don't work when matrices don't have a dtype):

obj_points = obj_points.astype('float32')
img_points = img_points.astype('float32')

Then I try again:

>>> cv2.calibrateCamera([obj_points], [img_points],size, camera_matrix, dist_coefs)
OpenCV Error: Bad argument 
(For non-planar calibration rigs the initial intrinsic matrix must be specified) 
in cvCalibrateCamera2, file ....

What?! A different error at least. But I did supply an initial intrinsic matrix!

So I go back to the documentation, and notice the flags parameter:

flags – Different flags that may be zero or a combination of the following values:

CV_CALIB_USE_INTRINSIC_GUESS cameraMatrix contains valid initial values of fx, fy, cx, cy that are optimized further

...

Aha, so I have to tell the function explicitly to use the initial guesses I provided:

cv2.calibrateCamera([obj_points], [img_points],size, camera_matrix.T, dist_coefs,
                    flags=cv2.CALIB_USE_INTRINSIC_GUESS)

Hurrah! It works!

(Moral of the story - read the OpenCV documentation carefully, and use the newest version (i.e. on opencv.itseez.com) if you're using the Python cv2 interface. Also, consult the examples in the samples/python2 directory to supplement the documentation. With these two things you should be able to work out most problems.)

Answer 2

After the help from mathematical.coffee I have got this 3d calibration to run.

import cv2
from cv2 import cv
import numpy as np

obj_points = [[-9.7,3.0,4.5],[-11.1,0.5,3.1],[-8.5,0.9,2.4],[-5.8,4.4,2.7],[-4.8,1.5,0.2],[-6.7,-1.6,-0.4],[-8.7,-3.3,-0.6],[-4.3,-1.2,-2.4],[-12.4,-2.3,0.9],[-14.1,-3.8,-0.6],[-18.9,2.9,2.9],[-14.6,2.3,4.6],[-16.0,0.8,3.0],[-18.9,-0.1,0.3],[-16.3,-1.7,0.5],[-18.6,-2.7,-2.2]]
img_points = [[993.0,623.0],[942.0,705.0],[1023.0,720.0],[1116.0,645.0],[1136.0,764.0],[1071.0,847.0],[1003.0,885.0],[1142.0,887.0],[886.0,816.0],[827.0,883.0],[710.0,636.0],[837.0,621.0],[789.0,688.0],[699.0,759.0],[768.0,800.0],[697.0,873.0]]

obj_points = np.array(obj_points,'float32')
img_points = np.array(img_points,'float32')

w = 1680
h = 1050
size = (w,h)

camera_matrix = np.zeros((3, 3),'float32')
camera_matrix[0,0]= 2200.0
camera_matrix[1,1]= 2200.0
camera_matrix[2,2]=1.0
camera_matrix[0,2]=750.0
camera_matrix[1,2]=750.0 

dist_coefs = np.zeros(4,'float32')

retval,camera_matrix,dist_coefs,rvecs,tvecs = cv2.calibrateCamera([obj_points],[img_points],size,camera_matrix,dist_coefs,flags=cv.CV_CALIB_USE_INTRINSIC_GUESS)

The only problem I have now is why is the dist_coefs vector is 5 elements long when returned from the calibration function. the documentation says " if the vector contains four elements, it means that K3=0". But in fact K3 is is used, no matter the length of dist_coefs (4 or 5). Furthermore I can't seem to get flag CV_CALIB_FIX_K3 to work, tied to use that flag to force K3 to be zero. cashes saying an integer is required. this could be because I don't know how to do multiple flags at once, I'm just doing this, flags = (cv.CV..., cv.CV...).

Just to compare, from the matlab camera cal routine the results are...
    Focal length: 2210. 2207.
    principal point: 781. 738.
    Distortions: 4.65e-2 -9.74e+0 3.9e-3 6.74e-3 0.0e+0
    Rotation vector: 2.36 0.178 -0.131
    Translation vector: 16.016 2.527 69.549

From this code,
    Focal length: 1647. 1629.
    principal point: 761. 711.
    Distortions: -2.3e-1 2.0e+1 1.4e-2 -9.5e-2 -172e+2
    Rotation vector: 2.357 0.199 -0.193
    Translation vector: 16.511 3.307 48.946

I think if I could figure out how to force k3=0, the rest of the values would align right up.