Detect and visualize differences between two images with OpenCV Python

Question

I have two images and would like to make it obvious where the differences are. I want to add color to the two images such that a user can clearly spot all the differences within a second or two.

For example, here are two images with a few differences:

leftImage.jpg:

rightImage.jpg:

My current approach to make the differences obvious, is to create a mask (difference between the two images), color it red, and then add it to the images. The goal is to clearly mark all differences with a strong red color. Here is my current code:

import cv2  # load images image1 = cv2.imread("leftImage.jpg") image2 = cv2.imread("rightImage.jpg")  # compute difference difference = cv2.subtract(image1, image2)  # color the mask red Conv_hsv_Gray = cv2.cvtColor(difference, cv2.COLOR_BGR2GRAY) ret, mask = cv2.threshold(Conv_hsv_Gray, 0, 255,cv2.THRESH_BINARY_INV |cv2.THRESH_OTSU) difference[mask != 255] = [0, 0, 255]  # add the red mask to the images to make the differences obvious image1[mask != 255] = [0, 0, 255] image2[mask != 255] = [0, 0, 255]  # store images cv2.imwrite('diffOverImage1.png', image1) cv2.imwrite('diffOverImage2.png', image1) cv2.imwrite('diff.png', difference) 

diff.png:

diffOverImage1.png

diffOverImage2.png

Problem with the current code: The computed mask shows some differences but not all of them (see for example the tiny piece in the upper right corner, or the rope thingy on the blue packet). These differences are shown only very lightly in the computed mask, but they should be clearly red like the other differences.

Input: 2 images with some differences.

Expected Output: 3 images: the two input images but with the differences highlighted (clearly highlighted in a configurable color), and a third image containing only the differences (the mask).

Answer 1

To visualize differences between two images, we can take a quantitative approach to determine the exact discrepancies between images using the Structural Similarity Index (SSIM) which was introduced in Image Quality Assessment: From Error Visibility to Structural Similarity. This method is already implemented in the scikit-image library for image processing. You can install scikit-image with pip install scikit-image.

Using the structural_similarity() function from scikit-image, it returns a score and a difference image, diff. The score represents the structural similarity index between the two input images and can fall between the range [-1,1] with values closer to one representing higher similarity. But since you're only interested in where the two images differ, the diff image is what we'll focus on. Specifically, the diff image contains the actual image differences with darker regions having more disparity. Larger areas of disparity are highlighted in black while smaller differences are in gray.

The gray noisy areas are probably due to .jpg lossy compression. We would obtain a cleaner result if we used a lossless compression image format. The SSIM score after comparing the two images show that they are very similar.

Image similarity 0.9198863419190031

Now we filter through the diff image since we only want to find the large differences between the images. We iterate through each contour, filter using a minimum threshold area to remove the gray noise, and highlight the differences with a bounding box. Here's the result.

To visualize the exact differences, we fill the contours onto a mask and on the original image.

from skimage.metrics import structural_similarity import cv2 import numpy as np  before = cv2.imread('left.jpg') after = cv2.imread('right.jpg')  # Convert images to grayscale before_gray = cv2.cvtColor(before, cv2.COLOR_BGR2GRAY) after_gray = cv2.cvtColor(after, cv2.COLOR_BGR2GRAY)  # Compute SSIM between two images (score, diff) = structural_similarity(before_gray, after_gray, full=True) print("Image similarity", score)  # The diff image contains the actual image differences between the two images # and is represented as a floating point data type in the range [0,1]  # so we must convert the array to 8-bit unsigned integers in the range # [0,255] before we can use it with OpenCV diff = (diff * 255).astype("uint8")  # Threshold the difference image, followed by finding contours to # obtain the regions of the two input images that differ thresh = cv2.threshold(diff, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1] contours = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) contours = contours[0] if len(contours) == 2 else contours[1]  mask = np.zeros(before.shape, dtype='uint8') filled_after = after.copy()  for c in contours:     area = cv2.contourArea(c)     if area > 40:         x,y,w,h = cv2.boundingRect(c)         cv2.rectangle(before, (x, y), (x + w, y + h), (36,255,12), 2)         cv2.rectangle(after, (x, y), (x + w, y + h), (36,255,12), 2)         cv2.drawContours(mask, [c], 0, (0,255,0), -1)         cv2.drawContours(filled_after, [c], 0, (0,255,0), -1)  cv2.imshow('before', before) cv2.imshow('after', after) cv2.imshow('diff',diff) cv2.imshow('mask',mask) cv2.imshow('filled after',filled_after) cv2.waitKey(0) 

Note: scikit-image version used is 0.18.1. In previous versions, the function was skimage.measure.compare_ssim but has been depreciated and removed in 0.18.1. According to the docs, the functionality still exists but is now under the new skimage.metrics submodule under different names. The new updated function is skimage.metrics.structural_similarity