How to find the empty squares in a chess board image?

Question

I am currently writing an image recognition script that makes a 2D array out of an image of a chess board for my chess project. However, I found it quite difficult to find which squares are empty:

So far, I have used the Canny edge detection on my image after applying the gaussian blur, which yielded the following results:

The code I used was:

sigma = 0.33
v = np.median(img) 
img = cv2.GaussianBlur(img, (7, 7), 2) # we use gaussian blur on the image to make it clear.
lower = int(max(0, (1.0 - sigma) * v)) # we find the lower threshold.
upper = int(min(255, (1.0 + sigma) * v)) # we find the higher threshold.
img_edge = cv2.Canny(img, 50, 50) # we use the canny function to edge canny the image.
cv2.imshow('question', img_edge) # we show the image.
cv2.waitKey(0)

(You may notice I did not use the threshold I got, that's because I found it inaccurate. If anyone has any tips I'd love them!)

Now, after doing these steps, I have tried many other things such as finding contours, Hough transform, etc. Yet I can't seem to figure out how to move on from that and actually find out whether a square is empty.

Any help is appreciated!

Answer 1

Assuming you have some kind of square shaped input image covering the whole chess board (as the example suggests), you can resize the image by rounding width and height to the next smaller multiple of 8. So, you can derive 64 equally sized tiles from your image. For each tile, count the number of unique colors. Set up some threshold to distinguish two classes (empty vs. non-empty square), maybe by using Otsu's method.

That'd be my code (half of that is simply visualization stuff):

import cv2
import matplotlib.pyplot as plt
import numpy as np
from skimage.filters import threshold_otsu


# Round to next smaller multiple of 8
# https://www.geeksforgeeks.org/round-to-next-smaller-multiple-of-8/
def round_down_to_next_multiple_of_8(a):
    return a & (-8)


# Read image, and shrink to quadratic shape with width and height of
# next smaller multiple of 8
img = cv2.imread('m0fAx.png')
wh = np.min(round_down_to_next_multiple_of_8(np.array(img.shape[:2])))
img = cv2.resize(img, (wh, wh))

# Prepare some visualization output
out = img.copy()
plt.figure(1, figsize=(18, 6))
plt.subplot(1, 3, 1), plt.imshow(img)

# Blur image
img = cv2.blur(img, (5, 5))

# Iterate tiles, and count unique colors inside
# https://stackoverflow.com/a/56606457/11089932
wh_t = wh // 8
count_unique_colors = np.zeros((8, 8))
for x in np.arange(8):
    for y in np.arange(8):
        tile = img[y*wh_t:(y+1)*wh_t, x*wh_t:(x+1)*wh_t]
        tile = tile[3:-3, 3:-3]
        count_unique_colors[y, x] = np.unique(tile.reshape(-1, tile.shape[-1]), axis=0).shape[0]

# Mask empty squares using cutoff from Otsu's method
val = threshold_otsu(count_unique_colors)
mask = count_unique_colors < val

# Some more visualization output
for x in np.arange(8):
    for y in np.arange(8):
        if mask[y, x]:
            cv2.rectangle(out, (x*wh_t+3, y*wh_t+3),
                          ((x+1)*wh_t-3, (y+1)*wh_t-3), (0, 255, 0), 2)
plt.subplot(1, 3, 2), plt.imshow(count_unique_colors, cmap='gray')
plt.subplot(1, 3, 3), plt.imshow(out)
plt.tight_layout(), plt.show()

And, that'd be the output:

As you can see, it's not perfect. One issue is the camera position, specifically that angle, but you already mentioned in the comments, that you can correct that. The other issue, as also already discussed in the comments, is the fact, that some pieces are placed between two squares. It's up to you, how to handle that. (I'd simply place the pieces correctly.)

----------------------------------------
System information
----------------------------------------
Platform:      Windows-10-10.0.19041-SP0
Python:        3.9.1
PyCharm:       2021.1.1
Matplotlib:    3.4.2
NumPy:         1.20.3
OpenCV:        4.5.2
scikit-image:  0.18.1
----------------------------------------