Image copied in ROI doesn't follow camera c++. How to fix this?

Question

I work on Windows7 x64 with opencv and Visual Studio 2010 on c++ language.

I created a project in which I show to my camera a rectangular area (call squared_surface). This area is recognized by tracing a rectangle with findSquare () and drawSquares () of opencv file squares.cpp. On this rectangle I create a ROI and there I copy an image (let's call copied_image)

My problem is that when I rotate squared_surface (in front of camera), copied_image does not follow it.

I think I need to use the functions getPerpective () and warpPerspective (), but I do not know how. Can anyone help me?

Here's the code:

int main(){
 vector<vector<Point> > squares;

 cv::VideoCapture cap(0);
 for (;;) {
    cv::Mat image;
    cap >> image;
    findSquares(image, squares);
    for (size_t i = 0; i < squares.size(); i++) {
    Rect rectangle = boundingRect(Mat(squares[i]));
                
    if((rectangle.width<=630)&& (rectangle.width >= 420) && (rectangle.height<= 490) &&(rectangle.height >= 250 )) {
        cv::Size dsize = Size(rectangle.width, rectangle.height);
        Mat img1 = imread("scacchiera.jpg");
    cv::resize(img1,img1,dsize,0,0, INTER_LINEAR);
        Rect roi (rectangle.x, rectangle.y,rectangle.width, rectangle.height);
        
    Mat imageRoi(image, roi);
       img1.copyTo(imageRoi);
       }
    } 
  drawSquares(image, squares);
  imshow("camera",image);
  if(waitKey(30) >= 0) break;
}
return 0;
}

Thanks!

EDIT. I was thinking of rotating Copied_image, so it follows Squared_surface, but I need to calculate the angle of rotation of the rectangle identified by the camera (drawn in red in the above images). Is there a way to calculate this angle?

Or how can I do so that Copied_image follows Squared_surface when I rotate squared_surface?

Help me, please!

Answer 1

I think I found the bug. Rect rectangle = boundingRect(Mat(squares[i])); This is where the problem is. You are creating the variable rectangle as a bounding rectangle of the coordinates in squares[i]. So your code always tries to find out the bounding rectangle and not the actual rectangle.

Instead of using a bounding rectangle, try using a rotated rectangle. Here is how to use it: http://www710.univ-lyon1.fr/~eguillou/documentation/opencv2/classcv_1_1_rotated_rect.html

The rotated rectangle RotatedRect (const Point2f &_center, const Size2f &_size, float _angle) requires the center point location, floating point angle and the size. Since you have all the coordinates I think you can use basic math and trigonometry to calculate the center and the angle on how your rectangle should be rotated/orientated. Let me know if this helps.