Finding properties of sloppy hand-drawn rectangles

Question

Image I'm working with:

I'm trying to find each of the boxes in this image. The results don't have to be 100% accurate, just as long as the boxes found are approximately correct in position/size. From playing with the example for square detection, I've managed to get contours, bounding boxes, corners and the centers of boxes.

There are a few issues I'm running into here:

1. bounding rectangles are detected for both the inside and the outside of the drawn lines.
2. some extraneous corners/centers are detected.
3. I'm not sure how to match corners/centers with the related contours/bounding boxes, especially when taking nested boxes into account.

Image resulting from code:

Here's the code I'm using to generate the image above:

import numpy as np import cv2 from operator import itemgetter from glob import glob def angle_cos(p0, p1, p2):     d1, d2 = (p0-p1).astype('float'), (p2-p1).astype('float')     return abs( np.dot(d1, d2) / np.sqrt( np.dot(d1, d1)*np.dot(d2, d2) ) ) def makebin(gray):     bin = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 5, 2)     return cv2.bitwise_not(bin) def find_squares(img):     img = cv2.GaussianBlur(img, (11, 11), 0)     squares = []     points = []`     for gray in cv2.split(img):         bin = makebin(gray)         contours, hierarchy = cv2.findContours(bin, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)         corners = cv2.goodFeaturesToTrack(gray,len(contours)*4,0.2,15)         cv2.cornerSubPix(gray,corners,(6,6),(-1,-1),(cv2.TERM_CRITERIA_MAX_ITER | cv2.TERM_CRITERIA_EPS,10, 0.1))         for cnt in contours:             cnt_len = cv2.arcLength(cnt, True)             if len(cnt) >= 4 and cv2.contourArea(cnt) > 200:                 rect = cv2.boundingRect(cnt)                 if rect not in squares:                     squares.append(rect)     return squares, corners, contours if __name__ == '__main__':     for fn in glob('../1 (Small).jpg'):         img = cv2.imread(fn)         squares, corners, contours = find_squares(img)         for p in corners:             cv2.circle(img, (p[0][0],p[0][3]), 3, (0,0,255),2)         squares = sorted(squares,key=itemgetter(1,0,2,3))         areas = []         moments = []         centers = []         for s in squares:             areas.append(s[2]*s[3])             cv2.rectangle( img, (s[0],s[1]),(s[0]+s[2],s[1]+s[3]),(0,255,0),1)         for c in contours:             moments.append(cv2.moments(np.array(c)))         for m in moments:             centers.append((int(m["m10"] // m["m00"]), int(m["m01"] // m["m00"])))         for cent in centers:             print cent             cv2.circle(img, (cent[0],cent[1]), 3, (0,255,0),2)         cv2.imshow('squares', img)         ch = 0xFF & cv2.waitKey()         if ch == 27:             break     cv2.destroyAllWindows()              

Answer 1

I suggest a simpler approach as a starting point. For instance, morphological gradient can serve as a good local detector of strong edges, and threshold on it tends to be simple. Then, you can remove too small components, which is relatively easy for your problem too. In your example, each remaining connected component is a single box, so the problem is solved in this instance.

Here is what you would obtain with this simple procedure:

The red points represent the centroid of the component, so you could grow another box from there that is contained in the yellow one if the yellow ones are bad for you.

Here is the code for achieving that:

import sys import numpy from PIL import Image, ImageOps, ImageDraw from scipy.ndimage import morphology, label  def boxes(orig):     img = ImageOps.grayscale(orig)     im = numpy.array(img)      # Inner morphological gradient.     im = morphology.grey_dilation(im, (3, 3)) - im      # Binarize.     mean, std = im.mean(), im.std()     t = mean + std     im[im < t] = 0     im[im >= t] = 1      # Connected components.     lbl, numcc = label(im)     # Size threshold.     min_size = 200 # pixels     box = []     for i in range(1, numcc + 1):         py, px = numpy.nonzero(lbl == i)         if len(py) < min_size:             im[lbl == i] = 0             continue          xmin, xmax, ymin, ymax = px.min(), px.max(), py.min(), py.max()         # Four corners and centroid.         box.append([             [(xmin, ymin), (xmax, ymin), (xmax, ymax), (xmin, ymax)],             (numpy.mean(px), numpy.mean(py))])      return im.astype(numpy.uint8) * 255, box   orig = Image.open(sys.argv[1]) im, box = boxes(orig)  # Boxes found. Image.fromarray(im).save(sys.argv[2])  # Draw perfect rectangles and the component centroid. img = Image.fromarray(im) visual = img.convert('RGB') draw = ImageDraw.Draw(visual) for b, centroid in box:     draw.line(b + [b[0]], fill='yellow')     cx, cy = centroid     draw.ellipse((cx - 2, cy - 2, cx + 2, cy + 2), fill='red') visual.save(sys.argv[3])