Fast Trapezoid to Rectangle for Video

Question

I want to transform a trapezoid area into a rectangle. Example Picture (not perfectly trapezoid, but you get the idea):

to this:

I already can mark the corners of the trapezoid area and use getPerspectiveTransform to calculate the correct Matrix for warpPerspective to transform the image.

Unfortunately this transformation is really slow. Using it on ~80% of the area of a 720p webcam stream results in a drop to ~5fps. I suspect this might be because warpPerspective allows more transformation than i need.

Is there a faster way to transform an image from a trapezoid to a rectangle? (preferably using OpenCV)

More information:

• warpAffine can be used to make affine transformations. It is faster than warpPerspective, but (if i got it correctly) you can only change the angle of a Parallelogramm.
• Somewhat related but without an answer: Trapezoid to Rectangle

A Minimum Working Example (not fully working) based on the answer of AldurDisciple using the first picture in my post.

#include <opencv2/core/core.hpp>  
#include <opencv2/highgui/highgui.hpp>
#include "opencv2/opencv.hpp"
using namespace cv;
int main() {

    //Mat img = imread("EQ9in.png");
    Mat img = imread("C:\\ss819729\\Aufnahmen\\arbeiten\\EQ9in.png");
    int height = img.rows;
    int width = img.cols;

    vector<Point2f> corners_rectangle, corners_trapezoid;

    corners_rectangle.push_back(Point2f(0, 0));
    corners_rectangle.push_back(Point2f(img.cols, 0));
    corners_rectangle.push_back(Point2f(img.cols, img.rows));
    corners_rectangle.push_back(Point2f(0, img.rows));

    corners_trapezoid.push_back(Point2f(35, 6));
    corners_trapezoid.push_back(Point2f(419, 55));
    corners_trapezoid.push_back(Point2f(404, 44));
    corners_trapezoid.push_back(Point2f(10, 477));

    Mat_<float> H_rectangle_to_trapezoid = cv::getPerspectiveTransform(corners_rectangle, corners_trapezoid);

    cv::Mat_<float> mapx_32f(height, width), mapy_32f(height, width);
    for(int y = 0; y<height; ++y) {
        float *buff_mapx = ((float*) mapx_32f.data)+y*width;
        float *buff_mapy = ((float*) mapy_32f.data)+y*width;
        for(int x = 0; x<width; ++x) {
            cv::Mat_<float> pt(3, 1);
            pt(0) = x;
            pt(1) = y;
            pt(2) = 1;
            pt = H_rectangle_to_trapezoid*pt;
            pt /= pt(2);
            buff_mapx[x] = pt(0);
            buff_mapy[x] = pt(1);
        }
    }
    cv::Mat map1_16u, map2_16u;
    cv::convertMaps(mapx_32f, mapy_32f, map1_16u, map2_16u, CV_16SC2);

    cv::Mat img_rectified;
    cv::remap(img, img_rectified, map1_16u, map2_16u, cv::INTER_LINEAR);

    namedWindow("Rectangle Image", CV_WINDOW_AUTOSIZE);
    while(waitKey(1)!='q') {
        cv::remap(img, img_rectified, map1_16u, map2_16u, cv::INTER_LINEAR);
        imshow("Rectangle Image", img_rectified);
    }
    return 1;
}

Answer 1

If the warping transformation is constant, there is a much faster way than using warpPerspective at each frame, using function remap (documentation link). This function can be used as follows.

First, at the begining of your program, compute the transformation maps, containing the (x,y) coordinates in source image for each pixel of the source image:

cv::Mat_<float> corners_rectangle, corners_trapezoid;
// TODO: fill corners_rectangle and corners_trapezoid
cv::Mat_<float> H_rectangle_to_trapezoid = cv::getPerspectiveTransform(corners_rectangle, corners_trapezoid);
cv::Mat_<float> mapx_32f(height,width), mapy_32f(height,width);
for(int y=0; y<height; ++y)
{
    float *buff_mapx=((float*)mapx_32f.data)+y*width;
    float *buff_mapy=((float*)mapy_32f.data)+y*width;
    for(int x=0; x<width; ++x)
    {
        cv::Mat_<float> pt(3,1);
        pt(0) = x;
        pt(1) = y;
        pt(2) = 1;
        pt = H_rectangle_to_trapezoid*pt;
        pt /= pt(2);
        buff_mapx[x] = pt(0);
        buff_mapy[x] = pt(1);
    }
}
cv::Mat map1_16u,map2_16u;
cv::convertMaps(mapx_32f,mapy_32f,map1_16u,map2_16u,CV_16SC2);
// Keep map1_16u & map2_16u, discard the rest

Then at each frame, you only need to do the interpolation using the remap function:

cv::Mat img_rectified;
cv::remap(img_src, img_rectified, map1_16u, map2_16u, cv::INTER_LINEAR);

Since the computation of the coordinate transformation is done offline, this is much faster than using warpPerspective repeatedly.