iOS Determine the corners of a Business Card in realtime

Question

I want to implement a business card detecting functionality like this app (https://scanbot.io). The camera should detect a business card and automatically take a picture of it (only the business card).

My idea was using BradLarson's GPUImage library, detect the corners (using the Harris corner detection algorithm), calculate the biggest rectangle with the corners obtained and crop the image contained inside the rectangle.

Here is my code:

 - (void)setupFilter {
    videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPreset640x480 cameraPosition:AVCaptureDevicePositionBack];

    filter = [[GPUImageHarrisCornerDetectionFilter alloc] init];
    [(GPUImageHarrisCornerDetectionFilter *)filter setThreshold:0.01f];
    [(GPUImageHarrisCornerDetectionFilter *)filter setSensitivity:0.5f];
    [(GPUImageHarrisCornerDetectionFilter *)filter setBlurRadiusInPixels:2.0f];

    [videoCamera addTarget:filter];
    videoCamera.runBenchmark = YES;
    GPUImageView *filterview = [[GPUImageView alloc] init];
    self.view=filterview;

    GPUImageCrosshairGenerator *crosshairGenerator = [[GPUImageCrosshairGenerator alloc] init];
    crosshairGenerator.crosshairWidth = 22.0;
    [crosshairGenerator forceProcessingAtSize:CGSizeMake(480.0, 640.0)];

    [(GPUImageHarrisCornerDetectionFilter *)filter setCornersDetectedBlock:^(GLfloat* cornerArray, NSUInteger cornersDetected, CMTime frameTime) {
        [crosshairGenerator renderCrosshairsFromArray:cornerArray count:cornersDetected frameTime:frameTime];
    }];

    GPUImageAlphaBlendFilter *blendFilter = [[GPUImageAlphaBlendFilter alloc] init];
    [blendFilter forceProcessingAtSize:CGSizeMake(480.0, 640.0)];
    GPUImageGammaFilter *gammaFilter = [[GPUImageGammaFilter alloc] init];
    [videoCamera addTarget:gammaFilter];
    [gammaFilter addTarget:blendFilter];

    [crosshairGenerator addTarget:blendFilter];
    [blendFilter addTarget:filterview];

    [videoCamera startCameraCapture];

}

The problem is I don't know how to adjust property the threshold and sensibility attributes to get the corners (now I'm getting the corners for all the objects in the image).

I also don't know how to work with this GLfloat* cornerArray.

I don't know if I am on the right way... any other ideas about how to implement this functionality or is there any existing library?

Thanks!

Answer 1

Read about Hough Transform. With it, you can detect lines. I would urge you to detect straight lines and then find four lines that are approximately at a right angle to each other and take the rectangle with the biggest area.

The steps would be:

1. Edge detection using Sobel filter.
2. Hough transform to find all straight lines in the image.
3. Look at all parallel lines and then all lines 90 degrees to those parallel line pairs, to find possible rectangles.
4. Pick the rectangle you like best. This could be by area, or by being best aligned to the phone, or you require that all edges are inside the visible camera image, or some other method.

Lastly: Computer Vision is hard... don't expect easy results.

Addendum

I should note that step 3 above is very simple, because the angle the lines take are simply one dimension of your Hough space. So parallel lines will have in this dimension equal values, and orthogonal lines will be shifted by pi or 90 degrees.