Why floating calculation and casting shows different result in debug and release configuration?

Question

This is really a weird bug to me and it took me long time to figure out what's happening. To simplify things and reproduce, just create a empty win32 console application using VS2005 and use this code in the main method:

float a = 411.00418f;
float b = 1.0f;
float c = 0.076279849f;
unsigned short result = (unsigned short)( (a-b)/c );
unsigned short result2 = (unsigned short)( (float)((a-b)/c) );
// Debug: 5374, 5375
// Release: 5374, 5374
printf("%d, %d\n", result, result2);  

Why result2 shows different value in debug/release mode?

Answer 1

In MSVC, the default floating-point mode is precise (/fp:precise). Meaning that the optimizer may do certain optimizations to improve accuracy or performance.

Try changing mode to strict (/fp:strict). This will make the compiler follow the strict floating-point rules on rounding and such.

(EDIT: strict (/fp:strict) doesn't seem to work in this case...)

If you look at the disassembly of the optimized build, you can see that the entire computation has been folded and optimized out.

push    5374                    ; 000014feH
push    5374                    ; 000014feH
push    OFFSET ??_C@_07MHMABKGB@?$CFd?0?5?$CFd?6?$AA@
call    DWORD PTR __imp__printf
add esp, 12                 ; 0000000cH

EDIT : This looks like a compiler optimizer bug to me.

Under strict (/fp:strict), the following code produces different results:

float a = 411.00418f;
float b = 1.0f;
float c = 0.076279849f;

unsigned short result1 = (unsigned short)((float)((a-b)/c));

float d = (float)((a-b)/c);
unsigned short result2 = (unsigned short)( d );

Output:

5374, 5375

Pulling out the (float)((a-b)/c) into a separate assignment should not affect the results under strict (/fp:strict).

---

I know one of the guys who work on the MSVC optimizer. I'll send a bug report to him.

Update:

Here's their response:

Hi Alex, thanks for this bug report. I’ll try to fix this for the upcoming VC++ release, but it might not make it.

FWIW, the bug doesn’t reproduce if you throw /arch:SSE2, and since we are enabling /arch:SSE2 by default for the next VC++ release (https://connect.microsoft.com/VisualStudio/feedback/details/688736/compiler-generates-sse-instructions-without-arch-sse).

So, the default behavior will show that this bug is fixed. But if you revert back to the old FP model (throw /arch:IA32) the bug may still be present.

Eric

So they've confirmed this as a bug.