Color calibration with color checker using using Root-Polynomial Regression not giving correct results

Question

For a quantification project, I am in need of colour corrected images which produce the same result over and over again irrespective of lighting conditions.

Every image includes a X-Rite color-checker of which the colors are known in matrix format:

Reference=[[170, 189, 103],[46, 163, 224],[161, 133, 8],[52, 52, 52],[177, 128, 133],[64, 188, 157],[149, 86, 187],[85, 85, 85],[67, 108, 87],[108, 60, 94],[31, 199, 231],[121, 122, 122], [157, 122, 98],[99, 90, 193],[60, 54, 175],[160, 160, 160],[130, 150, 194],[166, 91, 80],[70, 148, 70],[200, 200, 200],[68, 82, 115],[44, 126, 214],[150, 61, 56],[242, 243, 243]]

For every image I calculate the same matrix for the color card present as an example:

Actual_colors=[[114, 184, 137], [2, 151, 237], [118, 131, 55], [12, 25, 41], [111, 113, 177], [33, 178, 188], [88, 78, 227], [36, 64, 85], [30, 99, 110], [45, 36, 116], [6, 169, 222], [53, 104, 138], [98, 114, 123], [48, 72, 229], [29, 39, 211], [85, 149, 184], [66, 136, 233], [110, 79, 90], [41, 142, 91], [110, 180, 214], [7, 55, 137], [0, 111, 238], [82, 44, 48], [139, 206, 242]]

Then I calibrate the entire image using a color correction matrix which was derived from the coefficient from the input and output matrices:

for im in calibrated_img:
    im[:]=colour.colour_correction(im[:], Actual_colors, Reference, "Finlayson 2015")

The results are as follows:

Where the top image represents the input and the down image the output. Lighting plays a key role in the final result for the color correction, but the first two images on the left should generate the same output. Once the images become too dark, white is somehow converted to red.. I am not able to understand why.

I have tried to apply a gamma correction before processing with no success. The other two models Cheung 2004 and Vandermonde gave worse results, as did partial least squares. The images are pretty well corrected from the yellow radiating lamps, but the final result is not clean white, instead they have a blueish haze over the image. White should be white.. What can I do to further improve these results?

---

Edit 23-08-2020: Based on @Kel Solaar his comments I have made changes to my script to include the steps mentioned by him as follows

#Convert image from int to float
Float_image=skimage.img_as_float(img)

#Normalise image to have pixel values from 0 to 1
Normalised_image = (Float_image - np.min(Float_image))/np.ptp(Float_image)

#Decoded the image with sRGB EOTF
Decoded_img=colour.models.eotf_sRGB(Normalised_image)  

#Performed Finlayson 2015 color correction to linear data:
for im in Decoded_img:
    im[:]=colour.colour_correction(im[:], Image_list, Reference, "Finlayson 2015")

#Encoded image back to sRGB
Encoded_img=colour.models.eotf_inverse_sRGB(Decoded_img)  

#Denormalized image to fit 255 pixel values
Denormalized_image=Encoded_img*255

#Converted floats back to integers
Integer_image=Denormalised_image.astype(int)

This greatly improved image quality as can be seen below:

However, lighting/color differences between corrected images are unfortunately still present.

Raw images can be found here but due note that they are upside down.

Measured values of color cards in images:

IMG_4244.JPG
[[180, 251, 208], [62, 235, 255], [204, 216, 126], [30, 62, 97], [189, 194, 255], [86, 250, 255], [168, 151, 255], [68, 127, 167], [52, 173, 193], [111, 87, 211], [70, 244, 255], [116, 185, 228], [182, 199, 212], [102, 145, 254], [70, 102, 255], [153, 225, 255], [134, 214, 255], [200, 156, 169], [87, 224, 170], [186, 245, 255], [44, 126, 235], [45, 197, 254], [166, 101, 110], [224, 255, 252]]

IMG_4243.JPG
[[140, 219, 168], [24, 187, 255], [148, 166, 73], [17, 31, 53], [141, 146, 215], [42, 211, 219], [115, 101, 255], [33, 78, 111], [24, 118, 137], [63, 46, 151], [31, 203, 255], [67, 131, 172], [128, 147, 155], [61, 98, 255], [42, 59, 252], [111, 181, 221], [88, 168, 255], [139, 101, 113], [47, 176, 117], [139, 211, 253], [19, 78, 178], [12, 146, 254], [110, 60, 64], [164, 232, 255]]

IMG_4241.JPG
[[66, 129, 87], [0, 90, 195], [65, 73, 26], [9, 13, 18], [60, 64, 117], [20, 127, 135], [51, 38, 176], [15, 27, 39], [14, 51, 55], [21, 15, 62], [1, 112, 180], [29, 63, 87], [54, 67, 69], [20, 33, 179], [10, 12, 154], [38, 92, 123], [26, 81, 178], [58, 44, 46], [23, 86, 54], [67, 127, 173], [5, 26, 77], [2, 64, 194], [43, 22, 25], [84, 161, 207]]

IMG_4246.JPG
[[43, 87, 56], [2, 56, 141], [38, 40, 20], [3, 5, 6], [31, 31, 71], [17, 85, 90], [19, 13, 108], [7, 13, 20], [4, 24, 29], [8, 7, 33], [1, 68, 123], [14, 28, 46], [28, 34, 41], [6, 11, 113], [0, 1, 91], [27, 53, 83], [11, 44, 123], [32, 21, 23], [11, 46, 26], [32, 77, 115], [2, 12, 42], [0, 29, 128], [20, 9, 11], [49, 111, 152]]

Actual colors of color card (or reference) are given in the top of this post and are in the same order as values given for images.

---

Edit 30-08-2020, I have applied @nicdall his comments:

#Remove color chips which are outside of RGB range
New_reference=[]
New_Actual_colors=[]
for L,K in zip(Actual_colors, range(len(Actual_colors))):
    if any(m in L for m in [0, 255]):
        print(L, "value outside of range")
    else:
        New_reference.append(Reference[K])
        New_Actual_colors.append(Actual_colors[K])

In addition to this, I realized I was using a single pixel from the color card, so I started to take 15 pixels per color chip and averaged them to make sure it is a good balance. The code is too long to post here completely but something in this direction (don't judge my bad coding here):

for i in Chip_list:
    R=round(sum([rotated_img[globals()[i][1],globals()[i][0],][0],
        rotated_img[globals()[i][1]+5,globals()[i][0],][0],
        rotated_img[globals()[i][1]+10,globals()[i][0],][0],
        rotated_img[globals()[i][1],(globals()[i][0]+5)][0],
        rotated_img[globals()[i][1],(globals()[i][0]+10)][0],
        rotated_img[globals()[i][1]+5,(globals()[i][0]+5)][0],
        rotated_img[globals()[i][1]+10,(globals()[i][0]+10)][0]])/(number of pixels which are summed up))

The result was dissapointing, as the correction seemed to have gotten worse but it is shown below:

New_reference = [[170, 189, 103], [161, 133, 8], [52, 52, 52], [177, 128, 133], [64, 188, 157], [85, 85, 85], [67, 108, 87], [108, 60, 94], [121, 122, 122], [157, 122, 98], [60, 54, 175], [160, 160, 160], [166, 91, 80], [70, 148, 70], [200, 200, 200], [68, 82, 115], [44, 126, 214], [150, 61, 56]]

#For Image: IMG_4243.JPG:
New_Actual_colors= [[139, 218, 168], [151, 166, 74], [16, 31, 52], [140, 146, 215], [44, 212, 220], [35, 78, 111], [25, 120, 137], [63, 47, 150], [68, 132, 173], [128, 147, 156], [40, 59, 250], [110, 182, 222], [141, 102, 115], [48, 176, 118], [140, 211, 253], [18, 77, 178], [12, 146, 254], [108, 59, 62]]

#The following values were omitted in IMG_4243:
[23, 187, 255] value outside of range
[115, 102, 255] value outside of range
[30, 203, 255] value outside of range
[61, 98, 255] value outside of range
[88, 168, 255] value outside of range
[163, 233, 255] value outside of range

I have started to approach the core of the problem but I am not a mathematician, however the correction itself seems to be the problem.. This is the color correction matrix for IMG4243.jpg generated and utilized by the colour package:

CCM=colour.characterisation.colour_correction_matrix_Finlayson2015(New_Actual_colors, New_reference, degree=1 ,root_polynomial_expansion=True)
print(CCM)
[[ 1.10079803 -0.03754644  0.18525637]
 [ 0.01519612  0.79700086  0.07502735]
 [-0.11301282 -0.05022718  0.78838144]]

Based on what I understand from the colour package code the New_Actual_colors is converted with the CCM as follows:

Converted_colors=np.reshape(np.transpose(np.dot(CCM, np.transpose(New_Actual_colors))), shape)

When we compare the Converted_colors with the New_reference, we can see that the correction is getting a long way, but differences are still present (so the endgoal is to convert New_Actual_colors with the color correction matrix (CCM) to Converted_colors which should exactly match the New_reference):

print("New_reference =",New_reference)
print("Converted_colors =",Converted_colors)
New_reference =    [[170, 189, 103],[161, 133, 8],[52, 52, 52],[177, 128, 133],[64, 188, 157],[85, 85, 85],[67, 108, 87],[108, 60, 94],[121, 122, 122],[157, 122, 98],[60, 54, 175],[160, 160, 160],[166, 91, 80],[70, 148, 70],[200, 200, 200],[68, 82, 115],[44, 126, 214],[150, 61, 56]]
Converted_colors = [[176, 188, 106],[174, 140, 33],[26, 29, 38],[188, 135, 146],[81, 186, 158],[56, 71, 80],[48, 106, 99],[95, 50, 109],[102, 119, 122],[164, 131, 101],[88, 66, 190],[155, 163, 153],[173, 92, 70],[68, 150, 79],[193, 189, 173],[50, 75, 134],[55, 136, 192],[128, 53, 34]]

When substracted the differences become clear, and the question is how to overcome these differences?:

list(np.array(New_reference) - np.array(Converted_colors))
[array([-6,  1, -3]),
 array([-13,  -7, -25]),
 array([26, 23, 14]),
 array([-11,  -7, -13]),
 array([-17,   2,  -1]),
 array([29, 14,  5]),
 array([ 19,   2, -12]),
 array([ 13,  10, -15]),
 array([19,  3,  0]),
 array([-7, -9, -3]),
 array([-28, -12, -15]),
 array([ 5, -3,  7]),
 array([-7, -1, 10]),
 array([ 2, -2, -9]),
 array([ 7, 11, 27]),
 array([ 18,   7, -19]),
 array([-11, -10,  22]),
 array([22,  8, 22])]

Answer 1

Here are a few recommendations:

• As stated in my comment above we had an implementation issue with the Root-Polynomial variant from Finlayson (2015) which should be fixed in the develop branch.
• You are passing integer and encoded values to the colour.colour_correction definition. I would strongly recommend that you:
  • Convert the datasets to floating-point representation.
  • Scale it from range [0, 255] to range [0, 1].
  • Decode it with the sRGB EOTF.
  • Perform the colour correction onto that linear data.
  • Encode back and scale back to integer representation.
• Your images seem to be an exposure wedge, ideally, you would compute a single matrix for the appropriate reference exposure, normalise the other images exposure to it and apply the matrix on it.

Answer 2

An additional recommendation more on the physical side of the problem : I see some of the RGB values in the high and low exposure images are outside of the unsaturated range of the camera (0 and 255 values). This means that some information on the actual measured color is lost at the time of the image capture, because some of the calibration patches are either over- or under-exposed. This is a known problem in RGB colorimetry, and it is actually mentioned in (Finlayson, 2015) : "an additional assumption is that both v and kv are in the unsaturated range of the camera"

If possible, try to have a look at the histogram while you take the images so that all pixels have a value in the unsaturated range ([1, 254] at most).

Otherwise, if taking new images is out of the question, you can try ignoring the saturated patch (which have either 0 or 255 in any of R, G or B values) in the calibration process (make sure that you ignore the patches both in the image and in the reference). This could improve your calibration for the overall image as you do not make your model fit saturated values.