Multiple color object detection using OpenCV

Question

I am attempting to detect the position in the image of both the red walls and the white squares in the picture below of two white wall with red tops and white "posts":

My approach was to do thresholding to find the red walls which I can now easily detect from this output:

Now my problem is detecting the locations of the white squares, but this is more difficult considering the white walls. If I threshold based on white I still retain the undesired white walls in between the white square posts.

Any help would be greatly appreciated.

Answer 1

One approach consists in thresholding the input image with cv::inRange():

cv::Mat image = cv::imread(argv[1]);
if (image.empty())
{
    std::cout << "!!! Failed imread()" << std::endl;
    return -1;
}

cv::Mat red_image;
cv::inRange(image, cv::Scalar(40, 0, 180), cv::Scalar(135, 110, 255), red_image);
//cv::imwrite("out1.png", red_image);

Outputs:

We can use cv::findContours to retrieve the contours of the thresholded image to be able to create bounding boxes for them, which is a technique described here:

std::vector<std::vector<cv::Point> > contours;
std::vector<cv::Vec4i> hierarchy;
cv::findContours( red_image, 
                  contours, 
                  hierarchy, 
                  CV_RETR_TREE, 
                  CV_CHAIN_APPROX_SIMPLE, 
                  cv::Point(0, 0) );

std::vector<std::vector<cv::Point> > contours_poly( contours.size() );
std::vector<cv::Rect> boundRect( contours.size() );
for( int i = 0; i < contours.size(); i++ )
    { 
        cv::approxPolyDP( cv::Mat(contours[i]), contours_poly[i], 3, true );
        boundRect[i] = cv::boundingRect( cv::Mat(contours_poly[i]) );
    }   


// Debug purposes: draw bonding rects
//cv::Mat tmp = cv::Mat::zeros( red_image.size(), CV_8UC3 );
//for( int i = 0; i< contours.size(); i++ )
//  rectangle( tmp, boundRect[i].tl(), boundRect[i].br(), cv::Scalar(0, 255, 0), 2, 8, 0 );
//cv::imwrite("out2.png", tmp);  

Output:

All the rectangles displayed in the image above are stored as cv::Rect object inside the boundRect vector. Each rectangle is made of 2 opposite cv::Point objects, so we iterate on this vector to create a new vector made up of cv::Point objects only:

// Two opposite cv::Point can be used to draw a rectangle.
// Iterate on the cv::Rect vector and retrieve all cv::Point 
// and store them in a cv::Point vector.
std::vector<cv::Point> rect_points; 
for( int i = 0; i < contours.size(); i++ )
{
    rect_points.push_back(boundRect[i].tl());
    rect_points.push_back(boundRect[i].br());
}

//cv::Mat drawing = cv::Mat::zeros( red_image.size(), CV_8UC3 );
cv::Mat drawing = image.clone();

The logic to find the white squares is: assume that 2 pixels within 25x25 distance of each other define a white square:

// Draw a rectangle when 2 points are less than 25x25 pixels of 
// distance from each other
for( int i = 0; i < rect_points.size(); i++ )
{
    for( int j = 0; j < rect_points.size(); j++ )
    {
        if (i == j) 
            continue;

        int x_distance = (rect_points[i].x - rect_points[j].x);
        if (x_distance < 0) 
            x_distance *= -1;

        int y_distance = (rect_points[i].y - rect_points[j].y);
        if (y_distance < 0) 
            y_distance *= -1;

        if ( (x_distance < 25) && (y_distance < 25) )
        {
            std::cout << "Drawing rectangle " << i << " from " 
                      << rect_points[i] << " to " << rect_points[j] 
                      << " distance: " << x_distance << "x" << y_distance << std::endl;

            cv::rectangle( drawing, 
                           rect_points[i], 
                           rect_points[j], 
                           cv::Scalar(255, 50, 0), 
                           2 );
            break;
        }
    }

}

    //cv::imwrite("out3.png", drawing);
cv::imshow("white rectangles", drawing);    
cv::waitKey();

Output:

This algorithm is pretty raw and misses the 2 white squares at the bottom because there are no red walls below them, only above them.

So I leave it up to you to improve this approach :)

Good luck.

Answer 2

All the important information in the scene seems to be in the binarized picture of the extracted red bars. I would try to ignore the original picture for this step and use only the geometry of the rectangles to find the area between them.

For example, you could call findContours to obtain the 8 blobs in your example picture. If you check points on a line between their centers of mass, the point that returns a minimal value to pointPolygonTest is the center point of one of the white spots (or at least close).

You can use known information about the scene and the images of it you will encounter. For example, you could group the contours into "left" and "right" bars and only do the line search between certain contours. However, if you need to be more agnostic re. your input, you deriving more or less everything (scene orientation, number of walls, thickness of the rectangles...) from the thresholded picture should be quite possible.