How to ensure identical output files (*.o) using Google Protobufs in Linux

Question

I work in an industry that requires a repeatable build process when an application is ready for deployment. I've recently inherited an application that up until now has been in development. This application runs on a linux platform and uses Google Protocol Buffers (version 2.4.1). The build process is unfortunately not repeatable, but I've singled out the output files generated from the Protocol Buffer files as the source of uniqueness (using md5sum) in the application. The .o files are mostly the same with the exception of a few lines.

I repeated the process using the address book example distributed with the protobuf release. The differences in the .o files are listed below. Each time a .o file is generated the "878197C9XX" number will change. I'd like a repeatable process so they are always the same.

_GLOBAL__I__ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C918Person_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C918Person_reflection_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C930Person_PhoneNumber_reflection_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C923AddressBook_reflection_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C930protobuf_AssignDescriptorsOnceE
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C932protobuf_AssignDescriptors_once_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C923AddressBook_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C918Person_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C930Person_PhoneNumber_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C928Person_PhoneType_descriptor_E

Here is a second run:

_GLOBAL__I__ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E18Person_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E18Person_reflection_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E30Person_PhoneNumber_reflection_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E23AddressBook_reflection_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E30protobuf_AssignDescriptorsOnceEv
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E32protobuf_AssignDescriptors_once_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E23AddressBook_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E18Person_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E30Person_PhoneNumber_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E28Person_PhoneType_descriptor_E

The steps I used to create the .o files are as follows.

Starting with the addressbook.proto file in the protobuf-2.4.1/examples directory, the addressbook.pb.cc and adressbook.pb.h files are generated using ../bin/protoc --cpp_out=. addressbook.proto. The md5sum of the .cc and .h files are always the same.

When the .cc/.h files are compiled using g++ (gcc version 4.1.2 20080704, Red Hat 4.1.2-54), the output files are always unique. The command used for g++ is g++ -m32 -march=i686 -c -I ../src/ addressbook.pb.cc -o addressbook.o.

Any suggestions or ideas would be greatly appreciated. Thanks.

Answer 1

Those are C++ symbols in an anonymous namespace mangled by g++. The g++ compiler generates a unique name for each anonymous namespace. The unique namespace identifier is changing between each compilation.

Fortunately, you can use the -frandom-seed=string option to get a repeatable build. See On the way to deterministic binariy (gcc) output

Use the c++filt tool to de-mangle the symbols. You will see that both your 1st and 2nd compilation yield the same effective result:

$ head -n3 names.txt
_GLOBAL__I__ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C918Person_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C918Person_reflection_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_878197C930Person_PhoneNumber_reflection_E
$ head -n3 names2.txt
_GLOBAL__I__ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E18Person_descriptor_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E18Person_reflection_E
_ZN8tutorial46_GLOBAL__N_addressbook.pb.cc_00000000_75C23E1E30Person_PhoneNumber_reflection_E
$ c++filt < names.txt
global constructors keyed to tutorial::(anonymous namespace)::Person_descriptor_
tutorial::(anonymous namespace)::Person_reflection_
tutorial::(anonymous namespace)::Person_PhoneNumber_reflection_
tutorial::(anonymous namespace)::AddressBook_reflection_
tutorial::(anonymous namespace)::protobuf_AssignDescriptorsOnce
tutorial::(anonymous namespace)::protobuf_AssignDescriptors_once_
tutorial::(anonymous namespace)::AddressBook_descriptor_
tutorial::(anonymous namespace)::Person_descriptor_
tutorial::(anonymous namespace)::Person_PhoneNumber_descriptor_
tutorial::(anonymous namespace)::Person_PhoneType_descriptor_

$ c++filt < names2.txt
global constructors keyed to tutorial::(anonymous namespace)::Person_descriptor_
tutorial::(anonymous namespace)::Person_reflection_
tutorial::(anonymous namespace)::Person_PhoneNumber_reflection_
tutorial::(anonymous namespace)::AddressBook_reflection_
tutorial::(anonymous namespace)::protobuf_AssignDescriptorsOnce()
tutorial::(anonymous namespace)::protobuf_AssignDescriptors_once_
tutorial::(anonymous namespace)::AddressBook_descriptor_
tutorial::(anonymous namespace)::Person_descriptor_
tutorial::(anonymous namespace)::Person_PhoneNumber_descriptor_
tutorial::(anonymous namespace)::Person_PhoneType_descriptor_