How to align 2 images based on their content with OpenCV

Question

I am totally new to OpenCV and I have started to dive into it. But I'd need a little bit of help.

So I want to combine these 2 images:

I would like the 2 images to match along their edges (ignoring the very right part of the image for now)

Can anyone please point me into the right direction? I have tried using the findTransformECC function. Here's my implementation:

cv::Mat im1 = [imageArray[1] CVMat3];
cv::Mat im2 = [imageArray[0] CVMat3];

// Convert images to gray scale;
cv::Mat im1_gray, im2_gray;
cvtColor(im1, im1_gray, CV_BGR2GRAY);
cvtColor(im2, im2_gray, CV_BGR2GRAY);

// Define the motion model
const int warp_mode = cv::MOTION_AFFINE;

// Set a 2x3 or 3x3 warp matrix depending on the motion model.
cv::Mat warp_matrix;

// Initialize the matrix to identity
if ( warp_mode == cv::MOTION_HOMOGRAPHY )
    warp_matrix = cv::Mat::eye(3, 3, CV_32F);
else
    warp_matrix = cv::Mat::eye(2, 3, CV_32F);

// Specify the number of iterations.
int number_of_iterations = 50;

// Specify the threshold of the increment
// in the correlation coefficient between two iterations
double termination_eps = 1e-10;

// Define termination criteria
cv::TermCriteria criteria (cv::TermCriteria::COUNT+cv::TermCriteria::EPS,   number_of_iterations, termination_eps);

// Run the ECC algorithm. The results are stored in warp_matrix.
findTransformECC(
                 im1_gray,
                 im2_gray,
                 warp_matrix,
                 warp_mode,
                 criteria
                 );

// Storage for warped image.
cv::Mat im2_aligned;

if (warp_mode != cv::MOTION_HOMOGRAPHY)
    // Use warpAffine for Translation, Euclidean and Affine
    warpAffine(im2, im2_aligned, warp_matrix, im1.size(), cv::INTER_LINEAR + cv::WARP_INVERSE_MAP);
else
    // Use warpPerspective for Homography
    warpPerspective (im2, im2_aligned, warp_matrix, im1.size(),cv::INTER_LINEAR + cv::WARP_INVERSE_MAP);


UIImage* result =  [UIImage imageWithCVMat:im2_aligned];
return result;

I have tried playing around with the termination_eps and number_of_iterations and increased/decreased those values, but they didn't really make a big difference.

So here's the result:

What can I do to improve my result?

EDIT: I have marked the problematic edges with red circles. The goal is to warp the bottom image and make it match with the lines from the image above:

I did a little bit of research and I'm afraid the findTransformECC function won't give me the result I'd like to have :-(

Something important to add: I actually have an array of those image "stripes", 8 in this case, they all look similar to the images shown here and they all need to be processed to match the line. I have tried experimenting with the stitch function of OpenCV, but the results were horrible.

EDIT:

Here are the 3 source images:

The result should be something like this:

I transformed every image along the lines that should match. Lines that are too far away from each other can be ignored (the shadow and the piece of road on the right portion of the image)

Answer 1

By your images, it seems that they overlap. Since you said the stitch function didn't get you the desired results, implement your own stitching. I'm trying to do something close to that too. Here is a tutorial on how to implement it in c++: https://ramsrigoutham.com/2012/11/22/panorama-image-stitching-in-opencv/

Answer 2

You can use Hough algorithm with high threshold on two images and then compare the vertical lines on both of them - most of them should be shifted a bit, but keep the angle.

This is what I've got from running this algorithm on one of the pictures:

Filtering out horizontal lines should be easy(as they are represented as Vec4i), and then you can align the remaining lines together.

Here is the example of using it in OpenCV's documentation.

UPDATE: another thought. Aligning the lines together can be done with the concept similar to how cross-correlation function works. Doesn't matter if picture 1 has 10 lines, and picture 2 has 100 lines, position of shift with most lines aligned(which is, mostly, the maximum for CCF) should be pretty close to the answer, though this might require some tweaking - for example giving weight to every line based on its length, angle, etc. Computer vision never has a direct way, huh :)

UPDATE 2: I actually wonder if taking bottom pixels line of top image as an array 1 and top pixels line of bottom image as array 2 and running general CCF over them, then using its maximum as shift could work too... But I think it would be a known method if it worked good.