How to check if four points form a rectangle

Question

I am working on a shape recognition app. At this moment a set of points (x,y) is determined by corner detector (red points, img. 2.). Four of these points (in red frames, img. 2.) are vertices of a rectangle (sometimes a little deformed rectangle). What would be the best way to find them among others?

Here is an example of an input image:

And it looks like this after corner detection:

Answer 1

This is not an answer to your question - it's just suggestion.

In my opinion corner detector is a bad way to detect rectangles - it will take much time to calculate all point distances as mathematician1975 suggested. You have to use another technique in this situation:

1. That stamp is violet color, so first thing you should do is color segmentation.
2. After you've done with step 1 you can use Houhg transform to detect lines on binary image. Or find all contours in image.
3. And the final step is to detect rectangle.

Update:

Here's another solution that should also work in gray images.

1. Do a threshold to convert image to 1bit (I used 200 from 255 as threshold).
2. Find all contours in new image which have area bigger than some constant (I took 1000).
3. Find bounding rectangle for each contour and do a check:

ContourArea / BoundingReactangleArea > constant

I take this constant as 0.9.

And this algorithm gave me next result:

Here's OpenCV code:

Mat src = imread("input.jpg"), gray, result;
vector<vector<Point> > contours;
vector<Vec4i> hierarchy;

result = Mat(src.size(), CV_8UC1);

cvtColor(src, src, CV_BGR2GRAY);
threshold(src, gray, 200, 255, THRESH_BINARY_INV);
findContours(gray, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, Point(0, 0));

result = Scalar::all(0);
for (size_t i=0; i<contours.size(); i++)
{
    Rect rect = boundingRect(contours[i]);
    if (rect.area() > 1000)
    {
        double area = contourArea(contours[i]);
        if (area/rect.area() > 0.9)
        {
            drawContours(result, contours, i, Scalar(255), -1);
        }
    }
}