Python opencv detect shapes with intersections

Question

We are conducting shape detection using Python openCV. We find approx_poly_dp and then count number of vertices. However, the process does not work when object has many intersections (seen in second picture below). Opencv cannot detect individual objects, and just finds one large shape. In dealing with intersections, what is best way to find shapes, given general picture inputs. Or is this not a functionality of opencv, maybe better suited with machine learning?

    image_original = cv2.imread(file_name)
    image_gray = cv2.cvtColor(image_original, cv2.COLOR_BGR2GRAY)
    image_blur = cv2.GaussianBlur(image_gray, (5, 5), 0)
    image_morph = cv2.morphologyEx(image_blur, cv2.MORPH_OPEN, kernel)
    image_canny = cv2.Canny(image_morph, 50, 50)
    image_dilate = cv2.dilate(image_canny, kernel, iterations=2)
    image_final = cv2.erode(image_dilate, kernel, iterations=1)

    contours, hierarchy = cv2.findContours(image_final, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
    peri = cv2.arcLength(contours, True)
    approx = cv2.approxPolyDP(contours, 0.04 * peri, True)

    if len(approx) == 3:
        shape = "triangle"
    elif len(approx) == 4:
        (x, y, w, h) = cv2.boundingRect(approx)
        ar = w / float(h)
        shape = "square" if ar >= 0.95 and ar <= 1.05 else "rectangle"
    elif len(approx) == 5:
        shape = "pentagon"
    else:
        shape = "circle"

Intersection Picture:

Answer 1

Following procedure might work for simple figures:

Find the connected components, and find neighboring components of each component ignoring the background and outline components.

Try a component with combinations of its neighboring components and feed it to the shape classifier that you already have. As a pre-processing step, you might have to combine the components into a single blob using a morphological closing.

import cv2 as cv
import numpy as np
from itertools import combinations

im = np.zeros((512, 512), dtype=np.uint8)
im = cv.rectangle(im, (100, 200), (400, 400), 255, 2)
im = cv.circle(im, (250, 380), 100, 255, 2)
im = cv.rectangle(im, (50, 50), (250, 250), 255, 2)

ret, bw = cv.threshold(im, 0, 255, cv.THRESH_BINARY_INV | cv.THRESH_OTSU)
ncomp, labels, stats, centroids = cv.connectedComponentsWithStats(bw)

def getNeighbors(labels, n):
    r = 4 # you might have to change this depending on outline thickness
    se = cv.getStructuringElement(cv.MORPH_CROSS, (2*r+1, 2*r+1))
    neighbors = cv.dilate((labels==n).astype(np.uint8), se)
    # ignore background, outline and the component of interest
    return np.setdiff1d(np.unique(labels[neighbors==1]), [0, 1, n])

def shapes(arr, i):
    p = arr.tolist()
    for r in range(1, len(p)+1):
        for l in list(combinations(p, r)):
            print l + (i, )

For the following figure, you get the listed combinations of connected components. Note that I haven't bothered to remove duplicates.

for i in range(2, ncomp):
    shapes(getNeighbors(labels, i), i)

(3, 2)
(2, 3)
(4, 3)
(2, 4, 3)
(3, 4)
(5, 4)
(3, 5, 4)
(4, 5)
(6, 5)
(4, 6, 5)
(5, 6)

I wouldn't say this is an efficient solution as it involves trying out all combinations of neighbors, but I'm publishing this just in case someone finds this image processing based solution useful.