How do I remove the template from a form?

Question

Given the following two images:

Filled Form

Template

I would like to remove the template from this image, and leave behind ONLY the handwriting. I have code that aligns these images perfectly, but I am struggling on the code to remove the underlying template.

The code I currently have is as follows:

#Read in images and threshold
image = cv2.imread('image0.png')
template = cv2.imread('image1.png')
(thresh, im_bw) = cv2.threshold(image, 100, 255, cv2.THRESH_BINARY)
(thresh, temp_bw) = cv2.threshold(template, 100, 255, cv2.THRESH_BINARY)


#Convert temp from color to gray
graymask = cv2.cvtColor(temp_bw, cv2.COLOR_BGR2GRAY)

#Increase thickness of lines slightly
kernel = np.ones((2,2),np.uint8)
mask_crop = cv2.erode(graymask, kernel, iterations = 2)

(thresh, blackAndWhitemask) = cv2.threshold(mask_crop, 175, 255, cv2.THRESH_BINARY)
bw = cv2.bitwise_not(blackAndWhitemask)

#Inpaint
dst = cv2.inpaint(im_bw, bw, 3, cv2.INPAINT_NS)

The issue is that the resulting image Output does not look clean. You can clearly tell that there was a template there to begin with. Does anyone have any other techniques that they would reccomend?

Answer 1

The difference image solves most of the problem, but getting a clean signature is challenging.

First stage - finding where image and template are different:

# Read in images and threshold
image = cv2.imread('image0.png', cv2.IMREAD_GRAYSCALE)  # Read image as grayscale
template = cv2.imread('image1.png', cv2.IMREAD_GRAYSCALE)

diff = (image != template).astype(np.uint8)*255  # Find the difference and convert it to OpenCV mask format (255 where True).
cv2.imwrite('orig_diff.png', diff)

---

Small improvement:
Find where absolute difference is above 200:

thresh, diff = cv2.threshold(cv2.absdiff(image, template), 200, 255, cv2.THRESH_BINARY)

---

For covering the small black gaps (assuming all signatures are identical), we may use the following steps:

• Find contours in diff - each contour applies a signatures.
• Find bounding rectangles of all the signatures (of contours).
• Iterate each signature, and for each signature iterate all other signature and place the maximum of the two signatures.
  By putting the maximum value of two signatures, the black gaps are filled.
• Use the result of the previous stage as a mask.

The result is not perfect because the bounding boxes are not perfectly aligned, and because the original difference is too "thick".

---

Code sample:

import cv2
import numpy as np

#Read in images and threshold
image = cv2.imread('image0.png', cv2.IMREAD_GRAYSCALE)  # Read image as grayscale
template = cv2.imread('image1.png', cv2.IMREAD_GRAYSCALE)

#diff = (image != template).astype(np.uint8)*255  # Find the difference and convert it to OpenCV mask format (255 where True).
thresh, diff = cv2.threshold(cv2.absdiff(image, template), 200, 255, cv2.THRESH_BINARY)  # Find where absolute difference is above 200 and convert it to OpenCV mask format (255 where True).
cv2.imwrite('orig_diff.png', diff)

# Dilate diff for getting a gross place of the signatures
dilated_diff = cv2.dilate(diff, np.ones((51, 51), np.uint8))

# Find contours - each contour applies a signatures
cnts = cv2.findContours(dilated_diff, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)[0]

rects = []

# Find bounding rectangles of all the signatures
for c in cnts:
    bounding_rect = cv2.boundingRect(c)
    rects.append(bounding_rect)

# Cover black parts in diff - asuume all the signatures are the same.
# Iterate each signature, and for each signature iterate all other signature and place the maximum of the two signatures
for rect in rects:
    x1, y1, w1, h1 = rect
    for rect in rects:
        x2, y2, w2, h2 = rect
        w3 = min(w1, w2)
        h3 = min(h1, h2)
        roi1 = diff[y1:y1+h3, x1:x1+w3]
        roi2 = diff[y2:y2+h3, x2:x2+w3]
        diff[y2:y2+h3, x2:x2+w3] = np.maximum(roi1, roi2)

dst = image.copy()
dst[(diff == 0) | (image > 50)] = 255  # Place white color whrere diff=0 and also where image is white.

cv2.imwrite('diff.png', diff)
cv2.imwrite('dilated_diff.png', dilated_diff)
cv2.imwrite('dst.png', dst)

cv2.imshow('diff', diff)
cv2.imshow('dilated_diff', dilated_diff)
cv2.imshow('dst', dst)
cv2.waitKey()
cv2.destroyAllWindows()

---

Output:

---

orig_diff.png:

dilated_diff.png:

diff.png: