Cropping rectangular photos from scans in OpenCV with Python

Question

I have a bunch of photos like this:

I would like to automatically crop the image so that just the photograph (and possibly the caption) are shown.

I tried detecting contours, but they found the borders of objects in the photo and not the photo itself. There's also a spurious contour for the edges of the image as well as other small ones.

What can I do to get just the rectangle that includes the photo?

Answer 1

I managed to come up with a satisfactory solution to this. There are a few steps:

1. Get contours
2. Remove contours that are too small or too large in area
3. Find the min/max x/y over all remaining contours
4. Use those values to create a rectangle to crop in

That's the basic process.

Anyway, here's some code for the core parts:

import cv2
from os.path import basename
from glob import glob

def get_contours(img):
    # First make the image 1-bit and get contours
    imgray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    ret, thresh = cv2.threshold(imgray, 150, 255, 0)

    cv2.imwrite('thresh.jpg', thresh)
    img2, contours, hierarchy = cv2.findContours(thresh, 1, 2)

    # filter contours that are too large or small
    size = get_size(img)
    contours = [cc for cc in contours if contourOK(cc, size)]
    return contours

def get_size(img):
    ih, iw = img.shape[:2]
    return iw * ih

def contourOK(cc, size=1000000):
    x, y, w, h = cv2.boundingRect(cc)
    if w < 50 or h < 50: return False # too narrow or wide is bad
    area = cv2.contourArea(cc)
    return area < (size * 0.5) and area > 200

def find_boundaries(img, contours):
    # margin is the minimum distance from the edges of the image, as a fraction
    ih, iw = img.shape[:2]
    minx = iw
    miny = ih
    maxx = 0
    maxy = 0

    for cc in contours:
        x, y, w, h = cv2.boundingRect(cc)
        if x < minx: minx = x
        if y < miny: miny = y
        if x + w > maxx: maxx = x + w
        if y + h > maxy: maxy = y + h

    return (minx, miny, maxx, maxy)

def crop(img, boundaries):
    minx, miny, maxx, maxy = boundaries
    return img[miny:maxy, minx:maxx]

def process_image(fname):
    img = cv2.imread(fname)
    contours = get_contours(img)
    #cv2.drawContours(img, contours, -1, (0,255,0)) # draws contours, good for debugging
    bounds = find_boundaries(img, contours)
    cropped = crop(img, bounds)
    if get_size(cropped) < 400: return # too small
    cv2.imwrite('cropped/' + basename(fname), cropped)

process_image('pic.jpg')

This has the important parts, but I used two other tricks that worked well for my data set:

1. Modify the threshold until a certain percentage of the image is black. For most of my images even the lightest part of a photo is darker than the page underneath, so at a certain magic threshold level the photo becomes a black square and thus easier to get good contours for.

2. Completely ignore contours near the edges of the image. Sometimes a bit of the spine of the book causes contours to form at the borders of the original image, which is undesirable. Checking for contours within a small pixel count (like 20) of the edge and ignoring them solved that problem.

Some result images, original on left and autocropped on right: