Signalling NaN was corrupted when returning from x86 function (flds/fstps of x87)

Question

I have strange behaviour with x86 (32-bit) linux gcc. I generate signalling NaN using gcc's builtin __builtin_nansf(""), which generates 0x7fa00000. After returning this value from function as float, it is modified into 0x7fe00000. There is short example:

#include <stdio.h>
float f = __builtin_nansf("");
float y;

float func (void)
{
  return f;
}

int main (void)
{
  printf("%x\n", *((int*)&f));
  y = func();
  printf("%x\n", *((int*)&y));
}

Program compiled with gcc-4.6.2 program.c, its output:

7fa00000
7fe00000

Gdb:

(gdb) p/x f   
$2 = 0x7fa00000
...
(gdb) si
0x08048412 in func ()
1: x/i $pc
0x8048412 <func+14>:    flds   -0x4(%ebp)
(gdb) x/x $ebp-4
0xbfffeb34: 0x7fa00000
(gdb) si
(gdb) info all-regis
st0            nan(0xe000000000000000)  (raw 0x7fffe000000000000000)
... //after return from func
(gdb) si
0x0804843d in main ()
1: x/i $pc
0x804843d <main+38>:    fstps  0x804a024
(gdb) si
(gdb) x/x 0x804a024
0x804a024 <y>:  0x7fe00000

Why my signalling NaN is modified? How can I prevent this modification?

Answer 1

I'm not sure you can prevent this. Loading an sNaN on x87 typically raises an INVALID exception, and then converts the value to a qNaN, by setting the msb of the (23 bit) mantissa. That is, OR'ing with 0x00400000.

From the Intel® 64 and IA-32 Architectures Software Developer Manuals, Vol 1, 4.8.3.4 describes sNan/qNan handling. Chapter 8 deals with X87 FPU programming. Volume 3, 22.18 also describes how NaNs are handled by the X87 FPU.

I don't see any bits in the X87 control word that will yield the behaviour you desire for sNaN propagation.