Calculating homography matrix using arbitrary known geometrical relations

Question

I am using OpenCV for an optical measurement system. I need to carry out a perspective transformation between two images, captured by a digital camera. In the field of view of the camera I placed a set of markers (which lie in a common plane), which I use as corresponding points in both images. Using the markers' positions I can calculate the homography matrix. The problem is, that the measured object, whose images I actually want to transform is positioned in a small distance from the markers and in parallel to the markers' plane. I can measure this distance.

My question is, how to take that distance into account when calculating the homography matrix, which is necessary to perform the perspective transformation.

In my solution it is a strong requirement not to use the measured object points for calculation of homography (and that is why I need other markers in the field of view).

Please let me know if the description is not precise.

Presented in the figure is the exemplary image.

The red rectangle is the measured object. It is physically placed in a small distance behind the circular markers.

I capture images of the object from different camera's positions. The measured object can deform between each acquisition. Using circular markers, I want to transform the object's image to the same coordinates. I can measure the distance between object and markers but I do not know, how should I modify the homography matrix in order to work on the measured object (instead of the markers).

Answer 1

This question is quite old, but it is interesting and it might be useful to someone.

First, here is how I understood the problem presented in the question:

You have two images I₁ and I₂ acquired by the same digital camera at two different positions. These images both show a set of markers which all lie in a common plane p_m. There is also a measured object, whose visible surface lies in a plane p_o parallel to the marker's plane but with a small offset. You computed the homography H^m₁₂ mapping the markers positions in I₁ to the corresponding markers positions in I₂ and you measured the offset d_m-o between the planes p_o and p_m. From that, you would like to calculate the homography H^o₁₂ mapping points on the measured object in I₁ to the corresponding points in I₂.

A few remarks on this problem:

First, notice that an homography is a relation between image points, whereas the distance between the markers' plane and the object's plane is a distance in world coordinates. Using the latter to infer something about the former requires to have a metric estimation of the camera poses, i.e. you need to determine the euclidian and up-to-scale relative position & orientation of the camera for each of the two images. The euclidian requirement implies that the digital camera must be calibrated, which should not be a problem for an "optical measurement system". The up-to-scale requirement implies that the true 3D distance between two given 3D points must be known. For instance, you need to know the true distance l₀ between two arbitrary markers.

Since we only need the relative pose of the camera for each image, we may choose to use a 3D coordinate system centered and aligned with the coordinate system of the camera for I₁. Hence, we will denote the projection matrix for I₁ by P₁ = K₁ * [ I | 0 ]. Then, we denote the projection matrix for I₂ (in the same 3D coordinate system) by P₂ = K₂ * [ R₂ | t₂ ]. We will also denote by D₁ and D₂ the coefficients modeling lens distortion respectively for I₁ and I₂.

As a single digital camera acquired both I₁ and I₂, you may assume that K₁ = K₂ = K and D₁ = D₂ = D. However, if I₁ and I₂ were acquired with a long delay between the acquisitions (or with a different zoom, etc), it will be more accurate to consider that two different camera matrices and two sets of distortion coefficients are involved.

Here is how you could approach such a problem:

The steps in order to estimate P₁ and P₂ are as follows:

1. Estimate K₁, K₂ and D₁, D₂ via calibration of the digital camera

2. Use D₁ and D₂ to correct images I₁ and I₂ for lens distortion, then determine the marker positions in the corrected images

3. Compute the fundamental matrix F₁₂ (mapping points in I₁ to epilines in I₂) from the corresponding markers positions and infer the essential matrix E₁₂ = K₂^T * F₁₂ * K₁

4. Infer R₂ and t₂ from E₁₂ and one point correspondence (see this answer to a related question). At this point, you have an affine estimation of the camera poses, but not an up-to-scale one since t₂ has unit norm.

5. Use the measured distance l₀ between two arbitrary markers to infer the correct norm for t₂.

6. For the best accuracy, you may refine P₁ and P₂ using a bundle adjustment, with K₁ and ||t₂|| fixed, based on the corresponding marker positions in I₁ and I₂.

At this point, you have an accurate metric estimation of the camera poses P₁ = K₁ * [ I | 0 ] and P₂ = K₂ * [ R₂ | t₂ ]. Now, the steps to estimate H^o₁₂ are as follows:

7. Use D₁ and D₂ to correct images I₁ and I₂ for lens distortion, then determine the marker positions in the corrected images (same as 2. above, no need to re-do that) and estimate H^m₁₂ from these corresponding positions

8. Compute the 3x1 vector v describing the markers' plane p_m by solving this linear equation: Z * H^m₁₂ = K₂ * ( R₂ - t₂ * v^T ) * K₁^-1 (see HZ00 chapter 13, result 13.5 and equation 13.2 for a reference on that), where Z is a scaling factor. Infer the distance to origin d_m = ||v|| and the normal n = v / ||v||, which describe the markers' plane p_m in 3D.

9. Since the object plane p_o is parallel to p_m, they have the same normal n. Hence, you can infer the distance to origin d_o for p_o from the distance to origin d_m for p_m and from the measured plane offset d_m-o, as follows: d_o = d_m ± d_m-o (the sign depends of the relative position of the planes: positive if p_m is closer to the camera for I₁ than p_o, negative otherwise).

10. From n and d_o describing the object plane in 3D, infer the homography H^o₁₂ = K₂ * ( R₂ - t₂ * n^T / d_o ) * K₁^-1 (see HZ00 chapter 13, equation 13.2)

11. The homography H^o₁₂ maps points on the measured object in I₁ to the corresponding points in I₂, where both I₁ and I₂ are assumed to be corrected for lens distortion. If you need to map points from and to the original distorted image, don't forget to use the distortion coefficients D₁ and D₂ to transform the input and output points of H^o₁₂.

The reference I used:

[HZ00] "Multiple view geometry for computer vision", by R.Hartley and A.Zisserman, 2000.