Converting color value from float 0..1 to byte 0..255

Question

What would be the correct way of converting color value from float to byte? At first I thought b=f*255.0 should do it, but now I'm thinking, that in this case only the exact 1.0 will be converted to 255, but 0.9999 will already be 254 which is probably not what I want...

It seems that b=f*256.0 would be better except that it would have an unwanted case of making 256 in the case of exact 1.0.

In the end I'm using this:

#define F2B(f) ((f) >= 1.0 ? 255 : (int)((f)*256.0)) 

Answer 1

1.0 is the only case that can go wrong, so handle that case separately:

b = floor(f >= 1.0 ? 255 : f * 256.0) 

Also, it might be worth forcing that f really is 0<=f<=1 to avoid incorrect behaviour due to rounding errors (eg. f=1.0000001).

f2 = max(0.0, min(1.0, f)) b = floor(f2 == 1.0 ? 255 : f2 * 256.0) 

Alternative safe solutions:

b = (f >= 1.0 ? 255 : (f <= 0.0 ? 0 : (int)floor(f * 256.0))) 

or

b = max(0, min(255, (int)floor(f * 256.0))) 

Answer 2

I've always done round(f * 255.0).

There is no need for the testing (special case for 1) and/or clamping in other answers. Whether this is a desirable answer for your purposes depends on whether your goal is to match input values as closely as possible [my formula], or to divide each component into 256 equal intervals [other formulas].

The possible downside of my formula is that the 0 and 255 intervals only have half the width of the other intervals. Over years of usage, I have yet to see any visual evidence that that is bad. On the contrary, I've found it preferable to not hit either extreme until the input is quite close to it - but that is a matter of taste.

The possible upside is that [I believe] the relative values of R-G-B components are (slightly) more accurate, for a wider range of input values.
Though I haven't tried to prove this, that is my intuitive sense, given that for each component I round to get the closest available integer. (E.g. I believe that if a color has G ~= 2 x R, this formula will more often stay close to that ratio; though the difference is quite small, and there are many other colors that the 256 formula does better on. So it may be a wash.)

In practice, either 256 or 255-based approaches seem to provide good results.

---

Another way to evaluate 255 vs 256, is to examine the other direction -
converting from 0..255 byte to 0.0..1.0 float.

The formula that converts 0..255 integer values to equally spaced values in range 0.0..1.0 is:

f = b / 255.0 

Going in this direction, there is no question as to whether to use 255 or 256: the above formula is the formula that yields equally spaced results. Observe that it uses 255.

To understand the relationship between the 255 formulas in the two directions, consider this diagram, if you only had 2 bits, hence values integer values 0..3:

Diagram using 3 for two bits, analogous to 255 for 8 bits. Conversion can be from top to bottom, or from bottom to top:

0 --|-- 1 --|-- 2 --|-- 3   0 --|--1/3--|--2/3--|-- 1    1/6     1/2     5/6 

The | are the boundaries between the 4 ranges. Observe that in the interior, the float values and the integer values are at the midpoints of their ranges. Observe that the spacing between all values is constant in both representations.

If you grasp these diagrams, you will understand why I favor 255-based formulas over 256-based formulas.

---

Claim: If you use / 255.0 when going from byte to float, but you don't use round(f * 255.0) when going to byte from float, then the "average round-trip" error is increased. Details follow.

This is most easily measured by starting from float, going to byte, then back to float. For a simple analysis, use the 2-bit "0..3" diagrams.

Start with a large number of float values, evenly spaced from 0.0 to 1.0. THe round-trip will group all these values at the 4 values.
The diagram has 6 half-interval-length ranges:
0..1/6, 1/6..1/3, .., 5/6..1
For each range, the average round-trip error is half the range, so 1/12 (Minimum error is zero, maximum error is 1/6, evenly distributed).
All the ranges give that same error; 1/12 is the overall average error when round trip.

If you instead use any of the * 256 or * 255.999 formulas, most of the round-trip results are the same, but a few are moved to the adjacent range.
Any change to another range increases the error; for example if the error for a single float input previously was slightly less than 1/6, returning the center of an adjacent range results in an error slightly more than 1/6. E.g. 0.18 in optimal formula => byte 1 => float 1/3 ~= 0.333, for error |0.33-0.18| = 0.147; using a 256 formula => byte 0 => float 0 , for error 0.18, which is an increase from the optimal error 0.147.

Diagrams using * 4 with / 3. Conversion is from one line to the next.
Notice the uneven spacing of the first line: 0..3/8, 3/8..5/8, 5/8..1. Those distances are 3/8, 2/8, 3/8. Notice the interval boundaries of last line are different than first line.

   0------|--3/8--|--5/8--|------1          1/4     1/2     3/4 => 0------|-- 1 --|-- 2 --|------3    => 0----|---1/3---|---2/3---|----1        1/6       1/2       5/6 

The only way to avoid this increased error, is to use some different formula when going from byte to float. If you strongly believe in one of the 256 formulas, then I'll leave it to you to determine the optimal inverse formula.
(Per byte value, it should return the midpoint of the float values which became that byte value. Except 0 to 0, and 3 to 1. Or perhaps 0 to 1/8, 3 to 7/8! In the diagram above, it should take you from middle line back to top line.)

But now you will have the difficult-to-defend situation that you have taken equally-spaced byte values, and converted them to non-equally-spaced float values.

Those are your options if you use any value other than exactly 255, for integers 0..255: Either an increase in average round-trip error, or non-uniformly-spaced values in the float domain.