Feature detection - Small item in a large picture

Question

Assume you have two images. In one you have a small icon (like less than 300X300 pixels). The second is a very large one, and in within you have one (or multiple) smaller instances of the icon (of course at different scale, rotation).

The task at hand is to find the instances of the icon in the big image. How would you guys approach this?

I've tried to use a feature based object detection, by using the OpenCV library, however for crowded big images (containing many feature points), the matching is inconclusive. I've tried the SURF/OBJ feature extractors with the BRUTE/FLAN matching algorithms.

From my experience it seems like the matching takes no note of the geometrical relations between the feature points on the two images. As a visualization help I've attached an instance of the feature points of two example images.

And here's a harder instance of the task. I've highlighted the icon in the big image.

Answer 1

Based on limited experience, I have three recommendations for anyone else who runs into this type of issue.

1) Experiment with FindObject by Mathieu Labbé

This is a very nice tool that has helped me to experiment quickly to find good combinations of settings for feature detection / description. Just load your icon as an object and load your sample large images as scenes. Then tweak until you get reliable results for your application. As a bonus, he recently added the BRISK and FREAK non-patent-encumbered feature detection/description methods.

2) Get a realistic resolution

The resolution is so different for your icon object and actual icons in the large scenes that you are searching. In my understanding, the scale-invariance of these methods is actually pretty limited. Check out the excellent comparisons done by Ievgen Khvedchenia. You might get better results by resizing the image to the middle of the range that you expect it to be at.

3) Make the icon sample image more realistic (for example, blurred)

Related to #2, I have found that having a very sharp sample image when searching in a more realistic scene doesn't work well. I apply a gaussian to my sharp samples to make them more similar to what I expect to find. There is an example below. The strange formula just ensures that the kernel dimensions are odd numbers as required.

def proportional_gaussian(image):
    kernel_proportion = 0.005
    kernel_w = int(2.0 * round((image.shape[1]*kernel_proportion +1)/2.0)-1)
    kernel_h = int(2.0 * round((image.shape[0]kernel_proportion +1)/2.0)-1)
    return cv2.GaussianBlur(image, (kernel_w, kernel_h), 0)

Hope that helps someone.

Answer 2

Your feature matching approach is fine. If color can be used, use a preprocessing to find the region of interest such as histogram backprojection, followed by blob extraction and some shape analysis.

You should match all features from the big image to the small image. This results in many matches with few inliers.

As your icon is planar, you can use a homography as geometric contraint for your matches. Without writing a single line of code, you might try the OpenCV sample "descriptor_extractor_matcher" which can be found in the samples directory of the OpenCV sources.

./descriptor_extractor_matcher SURF SURF BruteForce NoneFilter icon.jpg image.jpg 3

See the help output of descriptor_extractor_matcher for further details.

You might try switching the order of the images, I don't remember which one is the training image and which one is the query image.

The paper on this kind of object detection is "Distinctive Image Features from Scale-Invariant Keypoints" by David Lowe. In section 7.3 of this paper he describes his approach of dealing with a very low inlier/outlier ratio which seems to be your case.

Good Luck!