How to Detect the bounds of a Passport page with OpenCV?

Question

I am trying to develop a scanner that can scan a page of a Passport with the camera.

So from a Passport page like this:

I'd like to crop out the marked part.

I have written code for edge detection using OpenCV which finds the contours and then approximates the largest quadrilateral. Finally it does a 4 point perspective transformation to get a top view of the image. The edge detection code look like this:

public static List<MatOfPoint> findContours(Mat src){
    Mat img = src.clone();
    src.release();
    //find contours
    double ratio = getScaleRatio(img.size());
    int width = (int) (img.size().width / ratio);
    int height = (int) (img.size().height / ratio);
    Size newSize = new Size(width, height);
    Mat resizedImg = new Mat(newSize, CvType.CV_8UC4);
    Imgproc.resize(img, resizedImg, newSize);

    Imgproc.medianBlur(resizedImg, resizedImg, 5);

    Mat cannedImg = new Mat(newSize, CvType.CV_8UC1);
    Imgproc.Canny(resizedImg, cannedImg, 70, 200, 3, true);
    resizedImg.release();

    Imgproc.threshold(cannedImg, cannedImg, 200, 255, Imgproc.THRESH_OTSU);

    Mat dilatedImg = new Mat(newSize, CvType.CV_8UC1);
    Mat morph = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(3, 3));
    Imgproc.dilate(cannedImg, dilatedImg, morph, new Point(-1, -1), 2, 1, new Scalar(1));
    cannedImg.release();
    morph.release();

    ArrayList<MatOfPoint> contours = new ArrayList<>();
    Mat hierarchy = new Mat();
    Imgproc.findContours(dilatedImg, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
    hierarchy.release();

    Log.d(TAG, "contours found: " + contours.size());

    Collections.sort(contours, new Comparator<MatOfPoint>() {
        @Override
        public int compare(MatOfPoint o1, MatOfPoint o2) {
            return Double.valueOf(Imgproc.contourArea(o2)).compareTo(Imgproc.contourArea(o1));
        }
    });

    return contours;
}

for(MatOfPoint contour:contours){
         MatOfPoint2f mat = new MatOfPoint2f(contour.toArray());
         double peri = Imgproc.arcLength(mat, true);
         MatOfPoint2f approx = new MatOfPoint2f();
         Imgproc.approxPolyDP(mat, approx, 0.02 * peri, true);

         Point[] points = approx.toArray();
         Log.d("SCANNER", "approx size " + points.length);

         if (points.length == 4) {
              Point[] spoints = CVProcessor.sortPoints(points);

              if (CVProcessor.insideArea(spoints, newSize)) {
                      rectContour = contour;
                      foundPoints = spoints;
                      break;
              }
        }
    }

this code works for single page documents i.e ID cards, credit cards. Where there are 4 distinguishable edges.

But doesn't work for passports as the top edge is not as distinctive.

The inputs will be taken from camera on Android. Any idea how can I detect the passport page? I am using OpenCV 3.1.

Here are a few sample inputs (obtained from Google image search):

Answer 1

It would be possible to extract the page if you can locate the Machine Readable Zone (MRZ) of the passport (region outlined in red in the image below). Usually, there's a very good contrast between the MRZ and its background, so it can be detected using a gradient based method or MSERs.

Assuming there's a standard template (that is, aspect ratios for the page, the MRZ, offsets for the fields, etc.) according to which the passport is prepared, once you locate the MRZ, it's easy to locate the page borders and other fields such as the photo of the person as shown in the template image below where the MRZ is outlined in red and the page border is outlined in green. This is assuming there's no perspective distortion. If there's such distortion, first you should correct it and then apply the template. You can use the MRZ itself to correct the distortion as you know the aspect ratio of the MRZ region.

Template prepared from image.

Check here for a very simple implementation of this template model based field extraction from a passport. It won't work for your images, and will need lots of parameter tuning, so I don't recommend using it straight away. I'm referring it just to convey the idea of template-based extraction and other pre-processing methods.

However, if the passport is curved as in the image below (you can see the MRZ boundary cannot be traced using straight lines), it is difficult to correct the distortion.

Finally, if you are using high resolution images, it would be a good idea to downsample and process them as it would be faster on an embedded system. Once you locate the MRZ from downsampled image, you can use the high-res image to refine the corners.