Hiding symbols in a shared library on Mac OS X

Question

We've been building a large open source software on a variety of platforms (Linux, Windows, Mac OS X, 32-bit and 64-bit) for several years without troubles. Lately however, the Mac OS X build (64-bit) stopped working correctly and started to crash randomly. It more or less coincided with an update of Mac OS X on our build machine from 10.7 to 10.8.2 (but the compiler toolchain didn't change, it's still llvm-gcc 4.2.1).

Our application is made of a couple of dynamic (shared) libraries and many executables using them. One of the shared library overrides the new and delete operators for a variety of reasons. On Mac OS X (and Linux), all symbols are exported by default, including our overloaded new and delete operators. The crashes on Mac OS X seem related to some memory being allocated with one memory subsystem (not ours) then freed through our own (and incompatible) delete implementation.

The sanest solution seems to be preventing the overloaded operators from being visible to the users of the shared library. This can be accomplished in two ways: marking the operators with __attribute__((visibility("hidden"))), or using the -unexported_symbols_list linker command line option to prevent some symbols from being exported. The first solution unfortunately doesn't work: gcc emits warnings saying that the operators have been declared differently (in <new>) and thus the attributes will be ignored. From my readings in various places, the second solution seems to be the right one to this problem. However for some reason we can't make it work.

When linking the shared library, we're passing the -Wl,-unexported_symbols_list unexported_symbols_list.txt option to g++, which in turns should be passed to ld. The unexported_symbols_list.txt file contains the following list of symbols:

__ZdaPv
__ZdaPvRKSt9nothrow_t
__ZdlPv
__ZdlPvRKSt9nothrow_t
__ZdlPvS_
__Znam
__ZnamRKSt9nothrow_t
__Znwm
__ZnwmPv
__ZnwmRKSt9nothrow_t

These are all the variations of new and delete that we override and want to be hidden. We found these symbols by doing nm libappleseed.dylib then unmangling the symbol names using c++filt.

Here's the command line generated by CMake to link libappeseed.dylib:

/usr/bin/g++  -g -Werror -dynamiclib -Wl,-headerpad_max_install_names -framework Cocoa -lcurl    -Werror -Wl,-unexported_symbols_list -Wl,unexported_symbols_list.txt -o ../mac-gcc4/appleseed/libappleseed.dylib [...]

Unfortunately, despite all our efforts it appears that the symbols remain (as nm shows).

Any idea what we are doing wrong? Is there another approach that we could try?

---

UPDATE Dec. 19, 2012:

Our problem and the supposed solution are well covered in this technical note from Apple: http://developer.apple.com/library/mac/#technotes/tn2185/_index.html (section "Overriding new/delete").

Pointers to relevant source code:

• operator new and operator delete overrides: allocator.cpp
• Macros to control symbol visibility in the shared library: dllvisibility.h

Fragment of nm's output after building libappleseed.dylib with -fvisibility=hidden and running strip -x libappleseed.dylib:

...
00000000002a41b0 T __ZdaPv
00000000002a41f0 T __ZdaPvRKSt9nothrow_t
00000000002a4190 T __ZdlPv
00000000002a41d0 T __ZdlPvRKSt9nothrow_t
00000000002a4060 T __Znam
00000000002a4130 T __ZnamRKSt9nothrow_t
00000000002a3ff0 T __Znwm
00000000002a40d0 T __ZnwmRKSt9nothrow_t
...

Answer 1

You should be building with -fvisibility=hidden and then export only what you want. Have a read here:

http://gcc.gnu.org/wiki/Visibility

It also explains -fvisibility-inlines-hidden. Many large libraries (Qt, for example) make use of this. The benefits are quite substantial.