How to isolate everything inside of a contour, scale it, and test the similarity to an image?

Question

I'm working on a project just for fun, and my goal is to play online poker and have the program identify the cards that are on the table. I am using OpenCV with python to isolate the area where the cards are going to be. I have been able to take an image of that area, grayscale and threshold it, and draw a contour around the edge of the card. I am now stuck on how to move forward.

This is my code so far:

import cv2
from PIL import ImageGrab
import numpy as np

def processed(image):
    grayscaled = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    thresholded = cv2.Canny(grayscaled, threshold1 = 200, threshold2 = 200)

    return thresholded

def drawcard1():
    screen = ImageGrab.grab(bbox = (770,300,850,400))

    processed_img = processed(np.array(screen))  

    outside_contour, dummy = cv2.findContours(processed_img.copy(), 0,2)

    colored = cv2.cvtColor(processed_img, cv2.COLOR_GRAY2BGR)

    cv2.drawContours(colored, outside_contour, 0, (0,255,0),2)     

    cv2.imshow('resized_card', colored)

while True:
    drawcard1()



    if cv2.waitKey(25) & 0xFF == ord('w'):
        cv2.destroyAllWindows()
        break

This is the result of what I have so far:

I need to be able to take the inside of the contour, and remove anything outside of it. Then the resulting image should only be the card, which I need to scale to 49x68 pixels. Once I can get that to work my plan is to get the contours of the rank and suit, and fill it in with white pixels, which I would then compare to a set of images to determine the best fit.

I am very new to OpenCV and image processing, but I find this stuff incredibly fascinating! I've been able to get this far with Google, but I can't find anything this time.

This is the image that I am using as a way to replace the game for now:

This is one of the images that I will use to compare the table cards to:

Answer 1

This situation is perfect for template matching. The idea is to search and find the location of a template image within a larger image. To perform this method, the template slides over the input image (similar to 2D convolution) where comparison methods are performed to determine pixel similarity. This is the basic idea behind template matching. Unfortunately, this basic method has flaws since it only works if the template image size is the same as the desired item to find in the input image. So if your template image was smaller than the desired region to find in the input image, this method would not work.

To get around this limitation, we can implement scale variant template matching by dynamically rescaling the image using np.linspace(). With each iteration, we resize the input image and keep track of the ratio. We continue resizing until the template image size is larger than the resized image while keeping track of the highest correlation value. A higher correlation value means a better match. Once we iterate through various scales, we find the ratio with the largest match and then compute the coordinates of the bounding box to determine the ROI.

---

Using your template image:

Here's the detected card highlighted in green. To visualize the process of dynamic template matching, uncomment the section in the code.

Code

import cv2
import numpy as np

# Resizes a image and maintains aspect ratio
def maintain_aspect_ratio_resize(image, width=None, height=None, inter=cv2.INTER_AREA):
    # Grab the image size and initialize dimensions
    dim = None
    (h, w) = image.shape[:2]

    # Return original image if no need to resize
    if width is None and height is None:
        return image

    # We are resizing height if width is none
    if width is None:
        # Calculate the ratio of the height and construct the dimensions
        r = height / float(h)
        dim = (int(w * r), height)
    # We are resizing width if height is none
    else:
        # Calculate the ratio of the 0idth and construct the dimensions
        r = width / float(w)
        dim = (width, int(h * r))

    # Return the resized image
    return cv2.resize(image, dim, interpolation=inter)

# Load template and convert to grayscale
template = cv2.imread('template.png')
template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
(tH, tW) = template.shape[:2]
cv2.imshow("template", template)

# Load original image, convert to grayscale
original_image = cv2.imread('1.jpg')
gray = cv2.cvtColor(original_image, cv2.COLOR_BGR2GRAY)
found = None

# Dynamically rescale image for better template matching
for scale in np.linspace(0.1, 3.0, 20)[::-1]:

    # Resize image to scale and keep track of ratio
    resized = maintain_aspect_ratio_resize(gray, width=int(gray.shape[1] * scale))
    r = gray.shape[1] / float(resized.shape[1])

    # Stop if template image size is larger than resized image
    if resized.shape[0] < tH or resized.shape[1] < tW:
        break

    # Threshold resized image and apply template matching
    thresh = cv2.threshold(resized, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]
    detected = cv2.matchTemplate(thresh, template, cv2.TM_CCOEFF)
    (_, max_val, _, max_loc) = cv2.minMaxLoc(detected)

    # Uncomment this section for visualization
    '''
    clone = np.dstack([thresh, thresh, thresh])
    cv2.rectangle(clone, (max_loc[0], max_loc[1]), (max_loc[0] + tW, max_loc[1] + tH), (0,255,0), 2)
    cv2.imshow('visualize', clone)
    cv2.waitKey(50)
    '''

    # Keep track of correlation value
    # Higher correlation means better match
    if found is None or max_val > found[0]:
        found = (max_val, max_loc, r)

# Compute coordinates of bounding box
(_, max_loc, r) = found
(start_x, start_y) = (int(max_loc[0] * r), int(max_loc[1] * r))
(end_x, end_y) = (int((max_loc[0] + tW) * r), int((max_loc[1] + tH) * r))

# Draw bounding box on ROI
cv2.rectangle(original_image, (start_x, start_y), (end_x, end_y), (0,255,0), 5)
cv2.imshow('detected', original_image)
cv2.imwrite('detected.png', original_image)
cv2.waitKey(0)