How do I crop an image using a binary mask image of the same picture to remove the background in python?

Question

I have tried to get the edge of the mask image with the following code:

import numpy as np
from matplotlib import pyplot as plt

img = cv2.imread('ISIC_0000000_segmentation.png',0)
edges = cv2.Canny(img,0,255)

plt.subplot(121), plt.imshow(img, cmap='gray')
plt.title('Original Image'), plt.xticks([]), plt.yticks([])
plt.subplot(122), plt.imshow(edges, cmap='gray')
plt.title('Edge Image'), plt.xticks([]), plt.yticks([])

plt.show

What I get is this:

But the edge isn't smooth for some reason.

My plan was to use the edge image to crop the following picture:

Does anyone know how I could make the edge image better and how I could use this to crop the normal image?

EDIT: @Mark Setchell made a good point: If i could use the mask image directly to crop the image that would be great.

Also: It is maybe possible to lay the normal image precisely on the mask image so that the black area on the mask would cover the blue-ish area on the normal picture.

EDIT: @Mark Setchell introduced the idea of multiplying the normale image with the mask image so what the background would result in 0(black) and the rest would keep its color. Would it be a problem when my mask image is .png and my normal picture is .jpg when multiplying?

EDIT: I have written the following code to try to multiply two pictures:

# Importing Image and ImageChops module from PIL package  
from PIL import Image, ImageChops 

# creating a image1 object 
im1 = Image.open("ISIC_0000000.jpg") 

# creating a image2 object 
im2 = Image.open("ISIC_0000000_segmentation.png") 

# applying multiply method 
im3 = ImageChops.multiply(im1, im2) 

im3.show()

But I get the error: ValueError: images do not match

Does anyone know how I could solve this?

Answer 1

If I understand correctly, you want to extract the object and remove the background. To do this, you can just do a simple cv2.bitwise_and() with the mask and the original input image.

Does anyone know how I could make the edge image better and how I could use this to crop the normal image?

To extract the background from the image, you don't need an edge image, the thresholded image can be used to remove only the desired parts of the image. You can use the mask image to directly drop the image and remove the background. Other approaches of obtaining a binary mask include using a fixed threshold value, adaptive threshold, or Canny edge detection. Here's a simple example using Otsu's threshold to obtain a binary mask followed by a bitwise-and operation.

Here's the result with the removed background

You can also turn all pixels on the mask to white if you wanted the removed background to be white

Note: Depending on how "smooth" you want the result, you can apply any blur to the image before thresholding to smooth out the edges. This can include averaging, Gaussian, median, or bilaterial filtering.

---

Code

import cv2

# Load image, grayscale, Otsu's threshold
image = cv2.imread('1.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

# Remove background using bitwise-and operation
result = cv2.bitwise_and(image, image, mask=thresh)
result[thresh==0] = [255,255,255] # Turn background white

cv2.imshow('thresh', thresh)
cv2.imshow('result', result)
cv2.waitKey()

Answer 2

The detected edge isn't smooth because the actual edge in the image isn't smooth. You can try filtering the original image first with low-pass filters.

If you can use contours, the following will work:

import numpy as np
import cv2
from matplotlib import pyplot as plt

# Read in image
imgRaw = cv2.imread('./Misc/edgesImg.jpg',0)

# Blur image
blurSize = 25
blurredImg = cv2.blur(imgRaw,(blurSize,blurSize))

# Convert to Binary
thrImgRaw, binImgRaw = cv2.threshold(imgRaw, 0, 255, cv2.THRESH_OTSU)
thrImgBlur, binImgBlur = cv2.threshold(blurredImg, 0, 255, cv2.THRESH_OTSU)

# Detect the contours in the image
contoursRaw = cv2.findContours(binImgRaw,cv2.RETR_TREE,cv2.CHAIN_APPROX_NONE)
contoursBlur = cv2.findContours(binImgBlur,cv2.RETR_TREE,cv2.CHAIN_APPROX_NONE)

# Draw all the contours
contourImgOverRaw = cv2.drawContours(imgRaw, contoursRaw[0], -1, (0,255,0),5)
contourImgOverBlur = cv2.drawContours(blurredImg, contoursBlur[0], -1, (0,255,0),5)

# Plotting
plt.figure()
plt.subplot(121)
plt.imshow(contourImgOverRaw)
plt.title('Raw Edges'), plt.xticks([]), plt.yticks([])
plt.subplot(122)
plt.imshow(contourImgOverBlur)
plt.title('Edges with {}px Blur'.format(blurSize)), plt.xticks([]), plt.yticks([])
plt.show()

EDIT: Here's more info on getting a mask of an image from contours.