Can I simply add affine or perspective (homography) matrices of transformation?

Question

As known, in OpenCV I can get affine or perspective transformation between 2 images:

• M - affine transformation - by using estimateRigidTransform()
• H - perspective (homography) transformation - by using FeatureDetector (SIFT, SURF, BRISK, FREAK, ...), then FlannBasedMatcher and findHomography()

Then I can do:

• affine transformation - by using warpAffine(img_src, img_dst, M)
• perspective transformation - by using warpPerspective(img_src, img_dst, H)

But if I have 3 or more images, and I already found:

• affine: M1 (img1 -> img2), M2 (img2 -> img3)
• perspective: H1 (img1 -> img2), H2 (img2 -> img3)

then can I get matix of transformation (img1 -> img3) by simply add two matrix?

• of an affine transform: M3 = M1 + M2;
• of an perspective transform: H3 = H1 + H2;

Or which of functions should I use for this?

Answer 1

No, you need to multiply the matrices to get the cascaded effect. I won't go into the math, but applying a transformation to coordinates is a matter of performing a matrix multiplication. If you are however curious as to know why that is, I refer you to this good Wikipedia article on cascading matrix transformations. Given a coordinate X and a transformation matrix M, you get the output coordinate Y by:

Y = M*X

Here I use * to refer to matrix multiplication as opposed to element-wise multiplication. What you have is a pair of transformation matrices which go from img1 to img2 then img2 to img3. You'll need to do the operation twice. So to go from img1 to img2 where X belongs to the coordinate space of img1, we have (assuming we're using the affine matrices):

Y1 = M1*X

Next, to go from img2 to img3, we have:

Y2 = M2*Y1 --> Y2 = M2*M1*X --> Y2 = M3*X --> M3 = M2*M1

Therefore, to get the desired chain effect, you need to create a new matrix such that M2 is multiplied by M1. Same as H2 and H1.

So define a new matrix such that:

cv::Mat M3 = M2*M1;

Similarly for your projective matrices, you can do:

cv::Mat H3 = H2*H1;

However, estimateRigidTransform (the output is M in your case) gives you a 2 x 3 matrix. One trick is to augment this matrix so that it becomes 3 x 3 where we add an additional row where it is all 0 except for the last element, which is set to 1. Therefore, you would have the last row such that it becomes [0 0 1]. You would do this for both matrices, multiply them, then extract just the first two rows into a new matrix to pipe into warpAffine. Therefore, do something like this:

// Create padded matrix for M1
cv::Mat M1new = cv::Mat(3,3,M1.type());
M1new.at<double>(0,0) = M1.at<double>(0,0);
M1new.at<double>(0,1) = M1.at<double>(0,1);
M1new.at<double>(0,2) = M1.at<double>(0,2);

M1new.at<double>(1,0) = M1.at<double>(1,0);
M1new.at<double>(1,1) = M1.at<double>(1,1);
M1new.at<double>(1,2) = M1.at<double>(1,2);

M1new.at<double>(2,0) = 0.0;
M1new.at<double>(2,1) = 0.0;
M1new.at<double>(2,2) = 1.0;

// Create padded matrix for M2
cv::Mat M2new = cv::Mat(3,3,M2.type());
M2new.at<double>(0,0) = M2.at<double>(0,0);
M2new.at<double>(0,1) = M2.at<double>(0,1);
M2new.at<double>(0,2) = M2.at<double>(0,2);

M2new.at<double>(1,0) = M2.at<double>(1,0);
M2new.at<double>(1,1) = M2.at<double>(1,1);
M2new.at<double>(1,2) = M2.at<double>(1,2);

M2new.at<double>(2,0) = 0.0;
M2new.at<double>(2,1) = 0.0;
M2new.at<double>(2,2) = 1.0;

// Multiply the two matrices together
cv::Mat M3temp = M2new*M1new;

// Extract out relevant rows and place into M3
cv::Mat M3 = cv::Mat(2, 3, M3temp.type());
M3.at<double>(0,0) = M3temp.at<double>(0,0);
M3.at<double>(0,1) = M3temp.at<double>(0,1);
M3.at<double>(0,2) = M3temp.at<double>(0,2);

M3.at<double>(1,0) = M3temp.at<double>(1,0);
M3.at<double>(1,1) = M3temp.at<double>(1,1);
M3.at<double>(1,2) = M3temp.at<double>(1,2);

When dealing with cv::Mat and the * operator, it is overloaded to specifically perform matrix multiplication.

You can then use M3 and H3 into warpAffine and warpPerspective respectively.

---

Hope this helps!