Detecting a broken shape using OpenCV

Question

I am trying to detect the square inside 4 bars. The goal is to compare their centroids. For the most part, my method (described below) correctly identifies the majority of cases. In images where the focus was bad or the lighting was too low, the processed image has a broken or disconnected square whose contour is not detected using Canny. Here is the general method flow:

1. Use template matching to find the 'bars' and create an ROI
2. Use adaptiveThreshold with ADAPTIVE_THRESH_MEAN_C on ROI
3. Isolate the square by setting the edges of the ROI to 255 (white)
4. Use morphologyEx with MORPH_OPEN and a (8,8) MORPH_RECT as the structuring element
5. Use an optimized Canny via the imutils.auto_canny method (written by Adrian Rosebrock at pyimagesearch.com) to find the square
6. Return the centers of the 'bars' and 'square' and display their minAreaRect's on the original image

Here is an example of a typical success: thresholded ROI, 'bars' removed, morphology applied, original with detected shapes

Here is an example of a broken, undetectable 'square': thresholded ROI, 'bars' removed, morphology applied, original with detected shapes

Things I've tried:

At step 2:

methods = ['cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)[1]',
           'cv2.threshold(cv2.GaussianBlur(gray, (5, 5), 0), 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]',
           'cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 51, 2)',
           'cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 51, 2)']

Global thresholding couldn't account for the varying brightnesses going from image to image. ADAPTIVE_THRESH_MEAN_C has a greater success rate than ADAPTIVE_THRESH_GAUSSIAN_C. The '51' in both of those was arbitrary; with testing it seemed like higher numbers produced a cleaner shape.

At step 4:

kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (8, 8))
roi = cv2.morphologyEx(roi, cv2.MORPH_OPEN, kernel)

For structuring elements, I've used RECT, ELLIPSE, and CROSS with various sizes. If they get too big, I end up creating shapes inside the 'square' that auto.canny prefers to detect and somehow can't find the outer 'square' anymore. Using RETR_EXTERNAL instead of RETR_LIST, RETR_CCOMP, or RETR_TREE does not fix this issue. For morphology operations, I've tried MORPH_OPEN and dilate/erode separately as a means to close the broken 'square' shape prior to contour detection.

At step 5:

auto_canny seems to work well. I've tried template matching the 'square' after all this processing, but it fails far too often.

I'm looking for a singular solution which captures both the current successes and some of the more "achievable" failures. I'm open to any wisdom you have to offer. New directions that I'm currently thinking about:

• Using line detection on the sides and find where they connect
• Could I approximate a square using approxPolyDP?
• Write my own detection algorithm just for the 'square'. It would involve finding the corners in the numpy array, creating a square between those points, and expanding it slightly.

While I hope that there is a magical method out there that does exactly what I'm looking for, I have spend several units of time working on this and expect a similarly difficult answer. Thank you in advance for your help!

Answer 1

Try filling in the square before finding the center. I did this by iterating through the rows and connecting the left-most and right-most black pixel.

# Iterate through each row in the image
for row in range(img.shape[0]):
    # Find the left-most and right-most pixel in the sqare
    start, stop = 0, 0
    for col in range(img.shape[1]):
        # Find the left-most
        if img[row,col] != 255 and start == 0: start = col, row
        # Find the right-most
        if img[row,col] != 255: stop = col, row
    # If there was a pixel in that row, connect them with a line
    if start != 0:
        cv2.line(img, start, stop, 0, 1)