How to check if obtained homography matrix is good?

Question

This question was already asked, but I still don't get it. I obtain a homography matrix by calling cv::findHomography from a set of points. I need to check whether it's relevant or not.
The proposed method is to calculate maximum reprojection error for inliers and compare it with a threshold. But after such filtration I keep getting insane transformations with object bounding box transforming to almost a straight line or some strange non-convex quadrangle, with self-intersections etc.
What constraints can be used to check if the homography matrix itself is adequate?

Answer 1

Your question is mathematical. Given a matrix of 3x3 decide whether it represents a good rigid transformation. It is hard to define what is "good" but here are some clues that can help you

1. Homography should preserve the direction of polygonal points. Design a simple test. points (0,0), (imwidth,0), (width,height), (0,height) represent a quadrilateral with clockwise arranged points. Apply homography on those points and see if they are still clockwise arranged if they become counter clockwise your homography is flipping (mirroring) the image which is sometimes still ok. But if your points are out of order than you have a "bad homography"
2. The homography doesn't change the scale of the object too much. For example if you expect it to shrink or enlarge the image by a factor of up to X, just check this rule. Transform the 4 points (0,0), (imwidth,0), (width-1,height), (0,height) with homography and calculate the area of the quadrilateral (opencv method of calculating area of polygon) if the ratio of areas is too big (or too small), you probably have an error.
3. Good homography is usually uses low values of perspectivity. Typically if the size of the image is ~1000x1000 pixels those values should be ~0.005-0.001. High perspectivity will cause enormous distortions which are probably an error. If you don't know where those values are located read my post: trying to understand the Affine Transform . It explains the affine transform math and the other 2 values are perspective parameters.

I think that if you check the above 3 condition (condition 2 is the most important) you will be able to detect most of the problems. Good luck