Python OpenCv intersection of multiple rectangles

Question

I am currently working on a project to recognise text sections in an image. I draw rectangles arround them as seen here:

and then cut them into each rectangles to do some more stuff with them. Like this:

What i do not want is, as seen in picture 2, to have a part of a rectangle be part of another one, eg. pic3 beeing part of pic2. That means these part should be cut out or made white, which i don't know how to do. For the drwaing of these rectangles i use this code section:

image = cv2.imread(Photopath)
CopyImg = np.copy(image)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, ((1, 1)), 0)
thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

# Create rectangular structuring element and dilate
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (6, 9))
dilate = cv2.dilate(thresh, kernel, iterations=4)

# Find contours and draw rectangle
cnts = cv2.findContours(dilate, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]

for c in cnts:
    x,y,w,h = cv2.boundingRect(c)
    cv2.rectangle(image, (x, y), (x + w, y + h), (36,255,12), 2)
    paragraph = CopyImg[y:y+h,x:x+w]

    cv2.imwrite(outpath, paragraph)
    

I guess i would need to somehow intersect them and cut out of one rectangle with the Coordinates of all other ones that touch it?

Here is the unprocessed image:

Answer 1

This is a possible way to do it. First, I’m gonna try to detect every block of text. It doesn’t matter if the blocks (or bounding boxes) overlap. After getting all bounding boxes for all the blobs on the image, I’ll detect bounding boxes overlaps. If a bounding box overlaps with other, this means that the same block of text will be shared among two or more images. I’ll crop the section and fill the overlapping area with a white rectangle, that way the content will only be shown on just one image.

These are the important steps:

1. Use morphology to get nice blocks of texts.

2. Detect contours on these blocks and convert those contours to bounding boxes.

3. Loop through all bounding boxes and:

   • Crop each bounding box

   • Check for possible overlaps. If an overlap (or intersection) is found, use the bounding box info to draw a white rectangle on this area.

This is the code, first we need to get those nice blocks of text:

import numpy as np
import cv2

# image path
path = "C:/opencvImages/"
fileName = "sheet05.png"

# Reading an image in default mode:
inputImage = cv2.imread(path + fileName)

# Some deep copies of the input mat:
inputCopy = inputImage.copy()
cleanInputCopy = inputCopy.copy()

# Grayscale conversion:
grayscaleImage = cv2.cvtColor(inputImage, cv2.COLOR_BGR2GRAY)

# Thresholding:
threshValue, binaryImage = cv2.threshold(grayscaleImage, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
binaryImage = 255 - binaryImage

Very standard stuff. Just read the image, convert it to grayscale and get a binary image via Otsu. You’ll notice I’ve created some deep copies of the input. These are used to mainly visualize the results, as I originally drew every bounding box found and every overlapping area.

Apply some very intense morphology to get the best possible blocks of text. I'm applying a dilate + erode here, with 10 iterations on each operation:

# Dilate and Erode with a big Structuring Element:
kernelSize = 5
structuringElement = cv2.getStructuringElement(cv2.MORPH_RECT, (kernelSize, kernelSize))
iterations = 10
dilatedImage = cv2.morphologyEx(binaryImage, cv2.MORPH_DILATE, structuringElement, None, None, iterations,
                                cv2.BORDER_REFLECT101)
erodedImage = cv2.morphologyEx(dilatedImage, cv2.MORPH_ERODE, structuringElement, None, None, iterations,
                               cv2.BORDER_REFLECT101)

The last snippet gets you this image:

Now, get the outer contours of this image and compute the bounding boxes:

# Find the big contours/blobs on the filtered image:
contours, hierarchy = cv2.findContours(erodedImage, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)

contours_poly = [None] * len(contours)
boundRect = []

# Alright, just look for the outer bounding boxes:
for i, c in enumerate(contours):
    if hierarchy[0][i][3] == -1:
        contours_poly[i] = cv2.approxPolyDP(c, 3, True)
        boundRect.append(cv2.boundingRect(contours_poly[i]))

Now, here comes the important part. We will loop through each bounding box. Crop the image, and check if the recently bounding box used for cropping the image is found inside another bounding box (an overlap). If that’s the case, just draw a big white rectangle inside that area and proceed with a new crop:

# Loop thru all bounding boxes:
for i in range(len(boundRect)):

    # Get current boundRect:
    sourceRect = boundRect[i]

    # Crop the roi:
    croppedImg = cleanInputCopy[sourceRect[1]:sourceRect[1] + sourceRect[3],
                 sourceRect[0]:sourceRect[0] + sourceRect[2]]

    # Check against other bounded rects:
    for j in range(len(boundRect)):

        # Get target boundRect:
        targetRect = boundRect[j]

        # Check for intersections:
        if i != j:

            foundIntersect, overlappedRect = checkIntersection(sourceRect, targetRect)

            if foundIntersect:

                # Found some overlapped rects, draw white rectangle at this location:
                cv2.rectangle(cleanInputCopy, (int(overlappedRect[0]), int(overlappedRect[1])),
                              (int(overlappedRect[0] + overlappedRect[2]), int(overlappedRect[1] + overlappedRect[3])),
                              (255, 255, 2550), -1)

    cv2.rectangle(inputCopy, (int(boundRect[i][0]), int(boundRect[i][1])),
                  (int(boundRect[i][0] + boundRect[i][2]), int(boundRect[i][1] + boundRect[i][3])), color, 5)

These are the bounding boxes detected for every binary blob:

The code detects the overlapping area and draws a white rectangle on the intersections, that way it will be no longer shown on the following crop:

These are the crops (Mind you, these are individual images):

Now, the helper function to detect bounding boxes intersections is this:

# Check for boxA and boxB intersection
def checkIntersection(boxA, boxB):
    
    x = max(boxA[0], boxB[0])
    y = max(boxA[1], boxB[1])
    w = min(boxA[0] + boxA[2], boxB[0] + boxB[2]) - x
    h = min(boxA[1] + boxA[3], boxB[1] + boxB[3]) - y

    foundIntersect = True
    if w < 0 or h < 0:
        foundIntersect = False

    return(foundIntersect, [x, y, w, h])

It is very straightforward. It just gets the coordinates of the two bounding rectangles and computes the intersection area. If the width or height are less than zero, there was no intersection.