Compare images SIFT /SURF details in .yml files OpenCV?

Question

I am java android developer, I don't know much about C/C++ or Matlab functions. What simple thing is am doing in my code is just create sift/Surf images details and save details in .yml files.

here is code how I am creating sift

vector < KeyPoint > keypoints;
Mat descriptors;
// Create a SIFT keypoint detector.
SiftFeatureDetector detector;
detector.detect(image, keypoints);
LOGI("Detected %d keypoints\n", (int) keypoints.size());
// Compute feature description.
detector.compute(image, keypoints, descriptors);

Saving the resulted descriptor in ( .yml ) file and later comparing that yml files using FlannBasedMatcher of OpenCV

here is my code

descriptors1 & descriptors2 is two mat object created from .yml files.

FlannBasedMatcher matcher;
vector<double> ee;
vector < DMatch > good_matches;
double good_matches_sum = 0.0;
vector < vector<DMatch> > matches;
matcher.knnMatch(descriptors1, descriptors2, matches, 2);
for (int i = 0; i < matches.size(); i++) {
   if (matches[i][0].distance < 0.8 * matches[i][1].distance) {

       good_matches.push_back(matches[i][0]);
       good_matches_sum += matches[i][0].distance;
    }
}

LOGI("good_matches_sum   %d s\n", good_matches_sum);
LOGI("the distance   k   %f s\n", good_matches_sum);

double score = (double) good_matches_sum / (double) good_matches.size();

LOGI("score   %f k   %d s\n", score, good_matches.size());

The problem is in above code is I am getting different result each time.e

here my two images as

//first time run for two images
good_matches_sum   1006632960 
the distance   k   3054.279755 
scores   254.523313 k   12 s


//Second time run for same two images
good_matches_sum   -402653184 
the distance   k   2835.513489 
score score scores   257.773954 k   11 s


//Third time run for same two images
good_matches_sum   -1946157056 
the distance   k   2794.588959 
score score scores   254.053542 k   11 s

I am assuming image slimier or dissimilar based on Positive result and negative. But with this kind of different result each time, i cant tell image similar or not.

please help me I don't know about Opencv and c so if any one has any idea then please suggest corrected code. thanks.

Answer 1

Basically, SIFT/SURF can't tell whether two images are similar or not. It can only tell you which key point in one image matches which corresponding point in the other image.

Luckily, the two function also provides you some more information, such as the 'distance' of a match, and the total number of matches in the image.

My idea of telling how similar the two image is :

1. Use findHomography() to get the homography of two images
2. Check whether the homography is valid (such as 4 of the points form a rectangle). 
3. Count the number of "good" matches (match.distance < SOME_DISTANCE)
4. Use some math to generate a score
5. If the previous homography is valid, increase the score

Here is the piece of code I once used, still there are some flaws in it (For some specific type of matches, the score is unreasonable. But this situation seldom occurs).

Notice, the MY_****_DISTANCE depend on whether you are using SIFT or SURF, and the parameters of the SIFT/SURF functions you give.

//===========SCORE ============
vector<double> matchDistance;
double avg = 0;
int avgCount = 0;
int goodCount = 0 ;

for( unsigned i = 0; i < matches.size(); i++ )
{
    double dist = matches[i].distance;

    //This is a "average match"
    if( dist < MY_AVG_DISTANCE  && dist > MY_LEAST_DISTANCE )
    {
        avg += dist;        //Count the average of distance of "average matches"
        avgCount++;
    }

    //This is a good match, and number of good match have a great impact on the result
    //Good matches are also average matches, that is, {GOOD_MATCH} is a subset of {AVERAGE_MATCH}
    if(dist < MY_GOOD_DISTANCE && dist > MY_LEAST_DISTANCE ){
        goodCount++;    //Count the number of "good matches"
    }
}

if(avgCount > 6){
    avg /= avgCount;    //Gives the average value
    if(goodCount < 12){ //If there are too few good matches, make a punishment
        avg = avg + (12-goodCount) * 4;
    }
}else{
    avg = MY_MAX_DISTANCE;
}

avg = avg > MY_AVG_DISTANCE ? MY_AVG_DISTANCE : avg;
avg = avg < MY_MIN_DISTANCE ? MY_MIN_DISTANCE : avg;


double score_avg = (MY_AVG_DISTANCE - avg) / ( MY_AVG_DISTANCE - MY_MIN_DISTANCE ) * 100;

if(homography_valid){   //If the previous homography is valid, make a reward. 
    score_avg += 40;
    score_avg = score_avg > 100 ? 100 : score_avg;
}else{
    score_avg -= 5;     //Or, make a little punishment
    score_avg = score_avg < 0 ? 0 : score_avg;
}

return score_avg

Update

My_****_DISTANCE

My_****_DISTANCE is something I defined myself, sorry for not explaining this before. This is the value I use, but you might want to change it to better fit your code.

#define MY_MIN_DISTANCE  200
#define MY_GOOD_DISTANCE 310
#define MY_AVG_DISTANCE 350
#define MY_MAX_DISTANCE 500
#define MY_MIN_HESSIAN 2000
#define MY_LEAST_DISTANCE 100

When running SIFT with parameters SiftFeatureDetector detector(400, 3, 0.04, 10.0, 1.6); , the result of some 'wrong' match compared to some 'right' match is:

Wrong   Right
307.993 330.124
470.419 307.374
219.775 371.026
294.389 400.696
355.321 259.239
331.926 189.042
222.317 457.089
320.718 379.061
423.09  201.95
371.098 200.646
362.427 343.229
441.167 359.32
253.253 382.15
367.191 215.678
405.19  358.686
390.251 343.798
341.905 238.002
341.073 226.519
363.775 262.5
340.742 174.877
320.214 415.802
249.405 195.261
347.357 328.76
343.839 116.331
351.058 383.93
286.224 111.472
352.976 138.701
298.409 238.044
385.34  223.716
264.571 331.115
333.339 208.103
329.588 128.168
372.971 267.83
331.804 222.578
301.935 232.459
351.504 342.524
300.762 379.87
346.872 390.031
374.281 308.198
304.746 401.452
307.184 193.298
229.943 98.0714
286.163 133.978
363.634 171.415
361.656 111.077
357.108 134.186
289.712 123.199
371.496 339.944
318.708 192.164
360.547 425.937
331.225 336.535
297.688 309.419
351.898 162.296
408.206 311.055
309.023 457.352
281.375 337.529
362.266 407.757
229.295 388.567
317.005 161.118
386.907 108.936
363.942 215.311
374.832 343.376
311.264 184.318
364.745 188.963
466.795 308.48
381.667 318.828
381.826 119.591
377.338 105.527
377.333 199.206
279.228 369.394
295.078 387.979
267.408 196.942
386.063 307.815
372.14  184.83
        294.927
        417.138
        348.458
        97.8621
        234.199
        144.094
        172.377
        131.412
        250.503
        227.139
        233.32
        116.258
        205.331
        354.505
        95.0368
        108.434
        116.46
        138.246
        406.135
        308.2
        92.817
        194.838
        312.103
        323.163
        312.946
        377.798
        359.393
        396.191
        320.272
        375.025
        309.383
        280.826
        278.456

It's obvious that the average of 'wrong' match is greater than the average of 'right' match. But if you run through many pictures, you will get to some exceptions (such as wrong matching with fewer matches but lower average distance ), some of which I haven't figured out how to eliminate. Thus, based on the raw data of matching distance, you also have to make a pretty 'personal' score.

findHomography

Here is a piece of demo you might need.

std::vector<Point2f> obj_corners(4);
obj_corners[0] = Point(0,0); 
obj_corners[1] = Point( img_object.cols, 0 );
obj_corners[2] = Point( img_object.cols, img_object.rows ); 
obj_corners[3] = Point( 0, img_object.rows );
std::vector<Point2f> scene_corners(4);
Mat H = findHomography( obj, scene, RANSAC );
perspectiveTransform( obj_corners, scene_corners, H);

And you can find detailed information in this tutorial

Update #2

Check_Valid_Homography

This depends on how you define "valid". Look at the official example, the picture in the left(the "object" image) is part of the picture in the right(the "scene" image). Then, apply the homography matrix to the vertex vectors of the "object" image, we finally get a green quadrangle appear in the right, which marks where the "object" image locates.

As for me, based on the tutorial, my definition of "valid" is: 1. The four points forms a quadrangle 2. None of the length of the side of the quadrangle should be too small 3. None of the angle in the quadrangle should be too small. 4. The quadrangle should not be up-side down.

And performing some basic maths, here is one part of the verification process.

bool RectChecker::check_order(const vector<Point2f> &corners, int height, int width){
    if(corners[0].y + 5 >= corners[3].y || corners[1].y + 5 >= corners[2].y){       //Point 0 should be above Point 3, so do Point1 and Point2
        return false;
    }
    if(corners[0].x + 5 >= corners[1].x || corners[3].x + 5 >= corners[2].x){       //Point 0 should be on the left of Point 1, so do Point3 and Point2
        return false;
    }
    int cnt = 0;
    for(int i = 0 ; i < corners.size() ; i++){          
        if(corners[i].x < -30 || corners[i].x > width + 30){
                cnt++;
                continue;
        }
        if(corners[i].y < -20 || corners[i].y > height + 20){               //This point is outside the image
                cnt++;
        }
    }
    if(cnt == 4){       //All points are outside the image
        return false;
    }
    return true;
}

Once again, you may want to create your own verification process, and as for the validation, it greatly depends on your requirements. But in my point, this part is the most simple one, while the most essential part is the scoring process.