Menu
Why does this assertion fail?
import std.conv;
void main()
{
auto y = 0.6, delta=0.1;
auto r = to!int(y/delta);
assert(r == 6);
}
r's value should be 6 and yet it's 5, Why?
927
On the Tools menu, click Options. In the Options dialog box, click the Error Checking tab. In the Settings section, click to select the Enable background error checking check box. In the Rules section, make sure the Number stored as text rule is selected, and then click OK.
Remove leading and trailing spaces around cells with numbers. Remove apostrophes in front of numbers. If the number format in the cells with numbers is "Text" then it will be changed to "General" in these cells.
In the Formula Bar, put the cursor in the cell which you want to make it constant, then press the F4 key. In this case, I don't want the cell reference A1 to be adjusted with the formula moving, so I put the cursor on A1 in the formula, and then press F4.
Next to the selected cell or range of cells, click the error button that appears. On the menu, click Convert to Number. (If you want to simply get rid of the error indicator without converting the number, click Ignore Error.) This action converts the numbers that are stored as text back to numbers.
This is probably because 0.6 can't be represented purely in a floating point number. You write 0.6, but that's not exactly what you get - you get something like 0.599999999. When you divide that by 0.1, you get something like 5.99999999, which converts to an integer of 5 (by rounding down).
Examples in other languages:
C#: Why is (double)0.6f > (double)(6/10f)?
Java: Can someone please explain me that in java why 0.6 is <0.6f but 0.7is >=0.7f
169
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
