GCC .obj file output is not deterministic (.debug_info, PROGBITS section)

Question

My compile command is
C:\work\PROJ-test\QNX_SDK\host\win32\x86/usr/bin/qcc -c -Wc,-frandom-seed="sadfsasafssadsa" -Wc,-MP,-MT,C:/work/PROJ-test/N_Manag/src/bld/N_Manag//armle-v7/release/nav_event_rcv.cpp.o,-MMD,C:/work/PROJ-test/N_Manag/src/bld/N_Manag//armle-v7/release/nav_event_rcv.cpp.d -Vgcc_ntoarmv7le -w9 -shared -O3 -ggdb3 -DBUILD_VERSION= -DPASLOGOPTIONS=0x02 -DPASLOGAPPZONES=31,23,30,9,8,3 -DNS1_5PORT -DBOARD_TYPE=PRODUCTION C:/work/PROJ-test/N_Manag/src/nav_event_rcv.cpp -o C:/work/PROJ-test/N_Manag/src/bld/N_Manag//armle-v7/release/nav_event_rcv.cpp.o

When I run this command twice in a row, the two .obj files are different and not just a few bytes from a timestamp.

We're switching build systems so we want our builds to be binary compatible. The vast majority of my object files are binary identical. A few that use the __DATE__ and __TIME__ macros are different by a few bytes but this one is wildly different!

I used an elf-dump utility and found the section that is wildly different between two compiles is this

  [544] .debug_info
       PROGBITS        00000000 047d70 1021ed 00   0   0  1
       [00000000]: 

But I don't know what PROGBITS contains and why it contains different items for consective compiles. This site just states that PROGBITS is an attribute but not what it indicates (and why it'd be different for consecutive compiles).

QUESTION

How do I make the generation of the .obj binary deterministic ?

THOUGHTS

Somehow, the code being compiled is actually modifying the .debug_info section of the .obj. This .cpp uses a bunch of boost libraries; is it possible that's the cause?

UPDATE

I looked at the assembly files being generated and they are different. Makes sense that the resulting .objs would be different.
Still doesn't make sense why this is happening.

UPDATE The qcc command above is not the actual compiler command executed: qcc is a compiler "redirector" in that it will call the one that matches the -V argument. The "real" compiler call is this:

C:/work/Proj/QNX_SDK/host/win32/x86/usr/lib/gcc/arm-unknown-nto-qnx6.5.0eabi/4.4.2/cc1plus -Wall -O3 -ggdb3 -DBUILD_VERSION= -DPASLOGOPTIONS=0x02 -DPASLOGAPPZONES=31,23,30,9,8,3 -DNS1_5PORT -DBOARD_TYPE=PRODUCTION -quiet -fno-builtin -fpic -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mlittle-endian -nostdinc -nostdinc++ -D__cplusplus -D__QNX__ -D__QNXNTO__ -D__GNUC__=4 -D__GNUC_MINOR__=4 -D__GNUC_PATCHLEVEL__=2 -D__NO_INLINE__ -D__DEPRECATED -D__EXCEPTIONS -D__unix__ -D__unix -D__ELF__ -fpic -DPIC=1 -D__ARM__ -D__arm__ -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp -D__LITTLEENDIAN__ -D__ARMEL__ -U__ARMEB__ -frandom-seed=sadfsasafssadsa -MP -MT C:/work/Proj/N_Manag/src/bld/N_Manag//armle-v7/release/nav_event_rcv.cpp.o -MMD C:/work/Proj/N_Manag/src/bld/N_Manag//armle-v7/release/nav_event_rcv.cpp.d -isystem C:/work/Proj/QNX_SDK/target/qnx6/usr/include -isystem C:/work/Proj/QNX_SDK/host/win32/x86/usr/lib/gcc/arm-unknown-nto-qnx6.5.0eabi/4.4.2/include -isystem C:/work/Proj/QNX_SDK/target/qnx6/usr/include/cpp/c -isystem C:/work/Proj/QNX_SDK/target/qnx6/usr/include/cpp C:/work/Proj/N_Manag/src/nav_event_rcv.cpp -dumpbase C:/work/Proj/N_Manag/src/nav_event_rcv.cpp -o C:\work\Proj\nav_event_rcv.s

UPDATE

I think it'd be worthwhile to look at the .s assembly output since there are major differences there.

Remember, I'm using -frandom-seed.

The .s file is 1.05mil lines and it's at line ~900k that the differences start.

Left:

.LASF17345:
.ascii "_ZN5boost6detail7variant21make_initializer_node5app"
.ascii "lyINS_3mpl4pairINS3_INS5_INS3_INS5_INS3_INS5_INS3_I"
.ascii "NS5_INS3_INS5_INS3_INS5_INS3_INS5_INS3_INS5_INS3_IN"
.ascii "S5_INS3_INS5_INS3_INS5_INS3_INS5_INS3_INS5_INS3_INS"
.ascii "5_INS3_INS5_INS3_INS5_INS3_INS5_INS3_INS5_INS1_16in"
.ascii "itializer_rootEN4mpl_4int_ILi0EEEEENS4_6l_iterINS4_"
...

Right:

.LASF17764:
.ascii "_ZNKSt8numpunctIcE13decimal_pointEv\000"
.LASF10304:
.ascii "cAlpha0\000"
.LASF10222:
.ascii "usWeek\000"
.LASF14117:
.ascii "_ZN5boost10shared_ptrI27TnRespTravelEstimationEvent"
.ascii "EaSERKS2_\000"
...

It goes on for several hundred bytes.

Now that I examine my beyond compare closely, all the difference sections are due to boost::detail::variant::make_initializer_node. Does that boost function generate different code each time?

RESOLUTION

Turns out it's a gcc bug. I compiled my .cpp with all permutations of -O<X> -ggdb<Y> and for Y>=2, the assembly files .s and the objects .obj are non-deterministic.

I found a gcc bug that describes this issue.

---

I had to delete the other post for . . . reasons.

Answer 1

Causes for non-determinism

The usual culprits are the macros __DATE__, __TIME__, __TIMESTAMP__ which the compiler expands to values calculated from the system time.

One possibility is that the debug info generated for the binary is written in a non-deterministic manner. This could happen, for example, when the in-memory layout of the debug info in the compiler process is not deterministic. I don't know the internals of GCC. But I guess something like this can happen when

• using random GUIDs in debug information output,
• using mangling of symbols in anonymous namespaces, or
• serializing a hashmap where the key is a pointer to memory, sequentially.

The latter source of non-determinism is usually considered to be a bug in the compiler (e.g. GCC PR65015)

Mitigation

To force reproducible expansions of the __DATE__, __TIME__ and __TIMESTAMP__ macros, one has to emulate and fake the system time (e.g. by using libfaketime/faketime) to the compiler. The -Wdate-time command-line option to GCC can be used to warn whenever these predefined macros are used.

To force reproducible "randomness" for GUIDs and mangling, you could try to compile with -frandom-seed=<somestring> where <somestring> is a unique string for your build (e.g. the hash of the contents of the source file you're compiling should do it).

Alternatively you can try to compile without debug information (e.g. without the -ggdb etc flags) or use some strip tool to remove the debug information section later.

See also

• https://wiki.debian.org/ReproducibleBuilds/About
• How to produce deterministic binary output with g++? - Stack Overflow