Static linking with generated protobufs causes abort

Question

I have a project which compiles c++ generated protobuf serializers into a static library. An executable links against this library, and an .so (.dll) does too. The executable later loads the .so file. When that happens, I get:

[libprotobuf ERROR /mf-toolchain/src/protobuf-3.0.0-beta-1/src/google/protobuf/descriptor_database.cc:57] File already exists in database: mri.proto
[libprotobuf FATAL /mf-toolchain/src/protobuf-3.0.0-beta-1/src/google/protobuf/descriptor.cc:1128] CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
what(): CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
Aborted (core dumped)

Just to be quite clear, I have one static library A, which is linked to program P and shared library S. Later, P loads S, and I get the error above.

I looked at similar errors on Stack Overflow and google in general, but I'm quite sure that I'm only linking against the library, not recompiling the source. As far as I can tell, this should mean that the compiled-in data is the same.

Also note: this problem only happens on Linux. This works fine on Windows and OS X.

Answer 1

The problem is that your static library contains a file mri.pb.cc which, in its global initializers, is registering type descriptors into the global descriptor database maintained by libprotobuf. Because your static library is loaded into your program twice, this initializer is running twice, but because you have only one copy of libprotobuf in your process, both initializers are registering into the same global database, and it is detecting a conflict.

To fix this problem, you need to change your static library into a shared library, which both the main program and the dynamically-loaded library depend on.

I am not sure why you see different behavior on Windows or OSX. My best guess is that on these platforms, you are actually linking two separate copies of libprotobuf into your program -- one in the main executable and one in the dynamically-loaded library. Thus there are two copies of the descriptor database and no conflict. However, you are likely to see much more subtle problems here. If you ever transfer protobuf object pointers between the main program and the dynamically-loaded module (without serializing and then parsing again) then you could end up having a protobuf object created by one copy of the library but being used with another copy (and therefore a different copy of the descriptor database) which will confuse the library and cause weird things to happen.

Alternatively, if you don't ever pass protobuf objects across the boundary, you might be able to "fix" the problem on Linux by linking libprotobuf statically, in order to get two copies as described above. But this is pretty dangerous; I wouldn't recommend it.