Using OpenCV to change the color temperature of a part of the image, using the rest of the image

Question

I have an image of a room, with an open door where we can see another room.
The white balance of the room where the picture is taken is correct but in the other room it's off because the illumination is different.
I would like to map the correct white from the original image (I'm using a mask) to the pixels in the other room (I also have a mask of those).

Here is my code:

cv::Mat blueMask;
cv::Mat whiteMask;
cv::Mat hsv;
getWhiteColorMask(src, blueMask, whiteMask, hsv);

// Save the HSV image
bool success = cv::imwrite(outputImagePath + "hsv.jpg", hsv);
// Save the white mask image
success = cv::imwrite(outputImagePath + "white_mask.jpg", whiteMask);
success = cv::imwrite(outputImagePath + "blue_mask.jpg", blueMask);
// Apply changes to the blue channel based on the mask
cv::Mat adjustedImage = src.clone();  // Clone the original image for adjustment

for (int row = 0; row < src.rows; ++row) {
    for (int col = 0; col < src.cols; ++col) {
        // Check if the pixel is in the blue mask
        if (blueMask.at<uchar>(row, col) > 0) {
            // Get the corresponding pixel in the white mask
            cv::Vec3b whitePixel = whiteMask.at<cv::Vec3b>(row, col);
        
            // Adjust the pixel in the blue mask using the white color
            cv::Vec3b& bluePixel = adjustedImage.at<cv::Vec3b>(row, col);
            bluePixel[0] = whitePixel[0];  // B channel
            //bluePixel[1] = whitePixel[1];  // G channel
            //bluePixel[2] = whitePixel[2];  // R channel
        }
    }
}
// Save the modified image with the new file name
std::string newImagePath = "./mask_test/white_masks/adjustedImage.jpg";
success = cv::imwrite(newImagePath, adjustedImage);

I have removed all the verification prints to make the code shorter for you so don't bother the bool success I'm using

Here are all the images needed to understand the problem:

Source image:

Incorrect white pixels mask:

Correct white pixels mask:

HSV image:

Answer 1

In general we have to correct the color ratios red/green and blue/green to match the rations in the other side (ratios of the "white" color).

There is no deterministic solution that is going to work for all cases, we have to use heuristics to find the above ratios in each side.

For finding the ratios of the white (gray) we may use White Balancing algorithm as Grayworld assumption.
In my solution I am using a variant of "max RGB" algorithm (assume high percentile of R,G,B represents a white pixel).

---

When applying color correction RGB values are supposed to be in linear space (without gamma).
First thing we should do is converting from sRGB to Linear RGB color space (inverting gamma).

We may also fix the mask for better represent the left and right sides (for testing purposes we may assume that we know how to split the left/right sides perfectly).

---

After inverting gamma, and fixing the mask, we may use the following stages:

• Extract Red, Green, Blue color channels from the RGB image.
• Create a mask of pixels that are not saturated ("burned") - we want to exclude the saturated pixels from the statistics.
• Extract all red, green, blue pixels in the left side (that are also not saturated) using the masks.
  Extract all red, green, blue pixels in the right side (that are also not saturated) using the masks.
• Compute upper percentile (say percentile 95) of R, G, B in the left side.
  Compute upper percentile (say percentile 95) of R, G, B in the left side.
  We assume that (leftRup, leftGup, leftBup) represents the white color in the left side.
  We assume that (rightRup, rightGup, rightBup) represents the white color in the right side.
  MATLAB code sample:

    leftRup = prctile(leftR, 95);
    leftGup = prctile(leftG, 95);
    leftBup = prctile(leftB, 95);

    rightRup = prctile(rightR, 95);
    rightGup = prctile(rightG, 95);
    rightBup = prctile(rightB, 95);

• Assume that the green pixels kept unmodified (assume that only the ratios are important).
  We want to correct the ratios of the left side in such way that:
  leftRup / leftGup = rightRup / rightGup
leftBup / leftGup = rightBup / rightGup
  Compute the scale factor of the red channel in the left side.
  Compute the scale factor of the blue channel in the left side.
  leftRscale = (rightRup/rightGup) / (leftRup/leftGup)
leftBscale = (rightBup/rightGup) / (leftBup/leftGup)

• Scale the red and blue channels of the left side (using the Mask).

• Join corrected R, G, B channels to into RGB image, and convert back from Linear RGB to sRGB (apply gamma)

---

MATLAB code sample (apologize for not using C++ or Python):

I = imread('original_image.jpg');
Mask = imread('mask.jpg');

figure;imshow(I);title(I);  % Show the input for testing

I = im2double(I); % Convert I from uint8 in range [0, 255] to double in range [0, 1] (use double for improving the accuracy).
I = rgb2lin(I); % Convert from sRGB to Linear RGB space (inverse gamma).

% Fix the mask, for getting better results.
Mask = imbinarize(Mask); % Convert to binary image.
Mask = bwareaopen(Mask, 5000); % Remove small connected components - assumed to be noise.
Mask = imdilate(Mask, ones(21, 21)); % Dilate the mask
Mask = bwfill(Mask, 'holes'); % Fill holes in the mask

% Extract Red, Green and Blue color channels
R = I(:, :, 1);
G = I(:, :, 2);
B = I(:, :, 3);

% Mask of non-saturated pixels (assume pixels values above 250 are saturated, and should be excluded).
NoSatR = R < 250;
NoSatG = G < 250;
NoSatB = B < 250;

leftR = R(Mask & NoSatR); % Extract all red pixels in the left side that are also not saturated.
leftG = G(Mask & NoSatG);
leftB = B(Mask & NoSatB);

rightR = R(~Mask & NoSatR); % Extract all red pixels in the right side that are also not saturated.
rightG = G(~Mask & NoSatG);
rightB = B(~Mask & NoSatB);

% Assume RGB of white pixel in the left size is (leftRup, leftGup, leftBup)
leftRup = prctile(leftR, 95); % Compute upper percentile of R in the left side.
leftGup = prctile(leftG, 95); % Compute upper percentile of G in the left side.
leftBup = prctile(leftB, 95); % Compute upper percentile of B in the left side.

% Assume RGB of white pixel in the right size is (rightRup, rightGup, rightBup)
rightRup = prctile(rightR, 95); % Compute upper percentile of R in the left side.
rightGup = prctile(rightG, 95); % Compute upper percentile of G in the left side.
rightBup = prctile(rightB, 95); % Compute upper percentile of B in the left side.

% We want to correct the ratios of the left side in such way that:
% leftRup / leftGup = rightRup / rightGup
% leftBup / leftGup = rightBup / rightGup
% We also assume that the green pixels kept unmodified (assume that only the ratios are important)

leftRscale = (rightRup/rightGup) / (leftRup/leftGup); % Compute the scale factor of the red channel in the left side
leftBscale = (rightBup/rightGup) / (leftBup/leftGup); % Compute the scale factor of the blue channel in the left side

% Scale the red and blue channels of the left side.
R(Mask) = R(Mask) * leftRscale;
B(Mask) = B(Mask) * leftBscale;

% Join corrected R, G, B channels to into RGB image
J = cat(3, R, G, B);

J = lin2rgb(J); % Convert from Linear RGB to sRGB (apply gamma).
J = im2uint8(J); % Convert back from double to uint8.

figure;imshow(J);title(J); % Show the output for testing

imwrite(J, 'J.jpg'); % Save the output for testing

---

Output:

---

Note:
Fixing the ratios of red/green and blue/green is an approximation.
There is a more accurate method named CAT - Chromatic Adaptation Correction.