Using opencv to find the most similar image that contains another image

Question

If the title isn't clear let's say I have a list of images (10k+), and I have a target image I am searching for.

Here's an example of the target image:

Here's an example of images I will want to be searching to find something 'similar' (ex1, ex2, and ex3):

Here's the matching I do (I use KAZE)

from matplotlib import pyplot as plt
import numpy as np
import cv2
from typing import List
import os
import imutils


def calculate_matches(des1: List[cv2.KeyPoint], des2: List[cv2.KeyPoint]):
    """
    does a matching algorithm to match if keypoints 1 and 2 are similar
    @param des1: a numpy array of floats that are the descriptors of the keypoints
    @param des2: a numpy array of floats that are the descriptors of the keypoints
    @return:
    """
    # bf matcher with default params
    bf = cv2.BFMatcher(cv2.NORM_L2)
    matches = bf.knnMatch(des1, des2, k=2)
    topResults = []
    for m, n in matches:
        if m.distance < 0.7 * n.distance:
            topResults.append([m])

    return topResults


def compare_images_kaze():
    cwd = os.getcwd()
    target = os.path.join(cwd, 'opencv_target', 'target.png')
    images_list = os.listdir('opencv_images')
    for image in images_list:
        # get my 2 images
        img2 = cv2.imread(target)
        img1 = cv2.imread(os.path.join(cwd, 'opencv_images', image))
        for i in range(0, 360, int(360 / 8)):
            # rotate my image by i
            img_target_rotation = imutils.rotate_bound(img2, i)

            # Initiate KAZE object with default values
            kaze = cv2.KAZE_create()
            kp1, des1 = kaze.detectAndCompute(img1, None)
            kp2, des2 = kaze.detectAndCompute(img2, None)
            matches = calculate_matches(des1, des2)

            try:
                score = 100 * (len(matches) / min(len(kp1), len(kp2)))
            except ZeroDivisionError:
                score = 0
            print(image, score)
            img3 = cv2.drawMatchesKnn(img1, kp1, img_target_rotation, kp2, matches,
                                      None, flags=2)
            img3 = cv2.cvtColor(img3, cv2.COLOR_BGR2RGB)
            plt.imshow(img3)
            plt.show()
            plt.clf()


if __name__ == '__main__':
    compare_images_kaze()

Here's the result of my code:

ex1.png 21.052631578947366
ex2.png 0.0
ex3.png 42.10526315789473

It does alright! It was able to tell that ex1 is similar and ex2 is not similar, however it states that ex3 is similar (even more similar than ex1). Any extra pre-processing or post-processing (maybe ml, assuming ml is actually useful) or just changes I can do to my method that can be done to keep only ex1 as similar and not ex3?

(Note this score I create is something I found online. Not sure if it's an accurate way to go about it)

ADDED MORE EXAMPLES BELOW

Another set of examples:

Here's what I am searching for

I want the above image to be similar to the middle and bottom images (NOTE: I rotate my target image by 45 degrees and compare it to the images below.)

Feature matching (as stated in answers below) were useful in found similarity with the second image, but not the third image (Even after rotating it properly)

Answer 1

Detecting The Most Similar Image

The Code

You can use template matching, where the image you want to detect if it's in the other images is the template. I have that small image saved in template.png, and the other three images in img1.png, img2.png and img3.png.

I defined a function that utilizes the cv2.matchTemplate to calculate the amount of confidence for if a template is in an image. Using the function on every image, the one that results ion the highest confidence is the image that contains the template:

import cv2

template = cv2.imread("template.png", 0)
files = ["img1.png", "img2.png", "img3.png"]

for name in files:
    img = cv2.imread(name, 0)
    print(f"Confidence for {name}:")
    print(cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED).max())

The Output:

Confidence for img1.png:
0.8906427
Confidence for img2.png:
0.4427919
Confidence for img3.png:
0.5933967

The Explanation:

1. Import the opencv module, and read in the template image as grayscale by setting the second parameter of the cv2.imread method to 0:

import cv2

template = cv2.imread("template.png", 0)

2. Define your list of images of which you want to determine which one contains the template:

files = ["img1.png", "img2.png", "img3.png"]

3. Loop through the filenames and read in each one as a grayscale image:

for name in files:
    img = cv2.imread(name, 0)

4. Finally, you can use the cv2.matchTemplate to detect the template in each image. There are many detection methods you can use, but for this I decided to use the cv2.TM_CCOEFF_NORMED method:

    print(f"Confidence for {name}:")
    print(cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED).max())

The output of the function ranges from between 0 and 1, and as you can see, it successfully detected that the first image is most likely to contain the template image (it has the highest level of confidence).

---

The Visualization

The Code

If detecting which image contains the template isn't enough, and you want a visualization, you can try the code below:

import cv2
import numpy as np

def confidence(img, template):
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
    res = cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED)
    conf = res.max()
    return np.where(res == conf), conf

files = ["img1.png", "img2.png", "img3.png"]

template = cv2.imread("template.png")
h, w, _ = template.shape

for name in files:
    img = cv2.imread(name)
    ([y], [x]), conf = confidence(img, template)
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 0, 255), 2)
    text = f'Confidence: {round(float(conf), 2)}'
    cv2.putText(img, text, (x, y), 1, cv2.FONT_HERSHEY_PLAIN, (0, 0, 0), 2)
    cv2.imshow(name, img)
    
cv2.imshow('Template', template)
cv2.waitKey(0)

The Output:

The Explanation:

1. Import the necessary libraries:

import cv2
import numpy as np

2. Define a function that will take in a full image and a template image. As the cv2.matchTemplate method requires grayscale images, convert the 2 images into grayscale:

def confidence(img, template):
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)

3. Use the cv2.matchTemplate method to detect the template in the image, and return the position of the point with the highest confidence, and return the highest confidence:

    res = cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED)
    conf = res.max()
    return np.where(res == conf), conf

4. Define your list of images you want to determine which one contains the template, and read in the template image:

files = ["img1.png", "img2.png", "img3.png"]
template = cv2.imread("template.png")

5. Get the size of the template image to later use for drawing a rectangle on the images:

h, w, _ = template.shape

6. Loop though the filenames and read in each image. Using the confidence function we defined before, get the x y position of the top-left corner of the detected template and the confidence amount for the detection:

for name in files:
    img = cv2.imread(name)
    ([y], [x]), conf = confidence(img, template)

7. Draw a rectangle on the image at the corner and put the text on the image. Finally, show the image:

    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 0, 255), 2)
    text = f'Confidence: {round(float(conf), 2)}'
    cv2.putText(img, text, (x, y), 1, cv2.FONT_HERSHEY_PLAIN, (0, 0, 0), 2)
    cv2.imshow(name, img)

8. Also, show the template for comparison:

cv2.imshow('Template', template)
cv2.waitKey(0)