Detecting a lego baseplate in an image

Question

I want my code to find the corners of a square lego plate in an image like the one attached.

I also want to find its dimensions, i.e. the number of "blops" in both dimensions (48x48 in the attached image).

I am currently looking at detecting the individual "blops", and the result so far is pretty good: a combination of blur, adaptiveThreshold, findContours and selection based on area finds the contours rendered in the second attached image (coloring is random).

I'm now looking for an algorithm to find the "grid" losely represented by these contours (or their mid-points), but I lack the google fu. Any ideas?

(Suggestions for different approaches are also very welcome.)

(The sample image shows bricks placed in the corners - an algorithm could expect this, if it helps.)

(The sample image has a rather wild background. I'd prefer to cope with that, if possible.)

Update 8 July 2016: I'm trying to write an algorithm that looks for streaks of adjacent contours forming lines. The algo should be able to find a number of these and, from that, deduce the form of the whole plate, even with perspective. Will update if it works...

Update December 2017: The above algorithm sort of worked, although it was a bit too unpredictable. Also I got problems with perspective (adding a "thick" lego brick changes the surface) and color recognition (shadows, camera peculiarities etc). This endeavor is on hold for now. If I resume it I will try with fixed camera positions immediately above the plate and consistent lights.

Answer 1

Here's a potential approach using color thresholding. The idea is to convert the image to HSV format then color threshold using a lower and upper bound with the assumption that the baseplate is in gray. This will give us a mask image. From here we morph open to remove noise, find contours, and sort for the largest contour. Next we obtain the rotated bounding box coordinates and draw this onto a new blank mask. Finally we bitwise-and the mask with the input image to get our result. To find the corner coordinates, we can use cv2.goodFeaturesToTrack() to find the points on the mask. Take a look at how to find accurate corner positions of a distorted rectangle from blurry image in python? and Shi-Tomasi Corner Detector & Good Features to Track for more details

---

Here's a visualization of each step:

We load the image, convert to HSV format, define a lower/upper bound, and perform color thresholding using cv2.inRange()

import cv2
import numpy as np

# Load image, convert to HSV, and color threshold
image = cv2.imread('1.png')
blank_mask = np.zeros(image.shape, dtype=np.uint8)
original = image.copy()
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower = np.array([0, 0, 109])
upper = np.array([179, 36, 255])
mask = cv2.inRange(hsv, lower, upper)

Next we create a rectangular kernel using cv2.getStructuringElement() and perform morphological operations using cv2.morphologyEx(). This step removes small particles of noise.

# Morph open to remove noise
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
opening = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel, iterations=1)

From here we find contours on the mask using cv2.findContours() and filter using contour area to obtain the largest contour. We then obtain the rotated boding box coordinates using cv2.minAreaRect() and cv2.boxPoints() then draw this onto a new blank mask with cv2.fillPoly(). This step gives us a "perfect" outer contour of the baseplate. Here's the detected outer contour highlighted in green and the resulting mask.

# Find contours and sort for largest contour
cnts = cv2.findContours(opening, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[0]

# Obtain rotated bounding box and draw onto a blank mask
rect = cv2.minAreaRect(cnts)
box = cv2.boxPoints(rect)
box = np.int0(box)
cv2.drawContours(image,[box],0,(36,255,12),3)
cv2.fillPoly(blank_mask, [box], (255,255,255))

Finally we bitwise-and the mask with our original input image to obtain our result. Depending on what you need, you can change the background to black or white.

# Bitwise-and mask with input image 
blank_mask = cv2.cvtColor(blank_mask, cv2.COLOR_BGR2GRAY)
result = cv2.bitwise_and(original, original, mask=blank_mask)
# result[blank_mask==0] = (255,255,255) # Color background white

To detect the corner coordinates, we can use cv2.goodFeaturesToTrack(). Here's the detected corners highlighted in purple:

Coordinates:

(91.0, 525.0)
(463.0, 497.0)
(64.0, 152.0)
(436.0, 125.0)

# Detect corners
corners = cv2.goodFeaturesToTrack(blank_mask, maxCorners=4, qualityLevel=0.5, minDistance=150)
for corner in corners:
    x,y = corner.ravel()
    cv2.circle(image,(x,y),8,(155,20,255),-1)
    print("({}, {})".format(x,y))

Full Code

import cv2
import numpy as np

# Load image, convert to HSV, and color threshold
image = cv2.imread('1.png')
blank_mask = np.zeros(image.shape, dtype=np.uint8)
original = image.copy()
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower = np.array([0, 0, 109])
upper = np.array([179, 36, 255])
mask = cv2.inRange(hsv, lower, upper)

# Morph open to remove noise
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
opening = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel, iterations=1)

# Find contours and sort for largest contour
cnts = cv2.findContours(opening, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[0]

# Obtain rotated bounding box and draw onto a blank mask
rect = cv2.minAreaRect(cnts)
box = cv2.boxPoints(rect)
box = np.int0(box)
cv2.drawContours(image,[box],0,(36,255,12),3)
cv2.fillPoly(blank_mask, [box], (255,255,255))

# Bitwise-and mask with input image 
blank_mask = cv2.cvtColor(blank_mask, cv2.COLOR_BGR2GRAY)
result = cv2.bitwise_and(original, original, mask=blank_mask)
result[blank_mask==0] = (255,255,255) # Color background white

# Detect corners
corners = cv2.goodFeaturesToTrack(blank_mask, maxCorners=4, qualityLevel=0.5, minDistance=150)
for corner in corners:
    x,y = corner.ravel()
    cv2.circle(image,(x,y),8,(155,20,255),-1)
    print("({}, {})".format(x,y))

cv2.imwrite('mask.png', mask)
cv2.imwrite('opening.png', opening)
cv2.imwrite('blank_mask.png', blank_mask)
cv2.imwrite('image.png', image)
cv2.imwrite('result.png', result)
cv2.waitKey()