Why are only some of these C++ template instantiations exported in a shared library?

Question

I have a C++ dynamic library (on macOS) that has a templated function with some explicit instantiations that are exported in the public API. Client code only sees the template declaration; they have no idea what goes on inside it and are relying on these instantiations to be available at link time.

For some reason, only some of these explicit instantiations are made visible in the dynamic library.

Here is a simple example:

// libtest.cpp
#define VISIBLE __attribute__((visibility("default")))

template<typename T> T foobar(T arg) {
    return arg;
}

template int  VISIBLE foobar(int);
template int* VISIBLE foobar(int*);

I would expect both instantiations to be visible, but only the non-pointer one is:

$ clang++ -dynamiclib -O2 -Wall -Wextra -std=c++1z -stdlib=libc++ -fvisibility=hidden -fPIC libtest.cpp -o libtest.dylib
$ nm -gU libtest.dylib | c++filt
0000000000000f90 T int foobar<int>(int)

This test program fails to link because the pointer one is missing:

// client.cpp
template<typename T> T foobar(T);  // assume this was in the library header

int main() {
    foobar<int>(1);
    foobar<int*>(nullptr);
    return 0;
}

$ clang++ -O2 -Wall -Wextra -std=c++1z -stdlib=libc++ -L. -ltest client.cpp -o client
Undefined symbols for architecture x86_64:
  "int* foobar<int*>(int*)", referenced from:
      _main in client-e4fe7d.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

There does seem to be some connection between the types and the visibility. If I change the return type to void, they are all visible (even if the template arguments are still pointers or whatever). Especially bizarre, this exports both:

template auto VISIBLE foobar(int)  -> int;
template auto VISIBLE foobar(int*) -> int*;

Is this a bug? Why would apparent syntactic sugar change behavior?

It works if I change the template definition to be visible, but it seems non-ideal because only a few of these instantiations should be exported... and I still want to understand why this is happening, either way.

I am using Apple LLVM version 8.0.0 (clang-800.0.42.1).

Answer 1

Your problem is reproducible on linux:

$ clang++ --version
clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
Target: x86_64-pc-linux-gnu
Thread model: posix

$ clang++ -shared -O2 -Wall -Wextra -std=c++1z -fvisibility=hidden \
-fPIC libtest.cpp -o libtest.so

$ nm -C libtest.so | grep foobar
0000000000000620 W int foobar<int>(int)
0000000000000630 t int* foobar<int*>(int*)

The non-pointer overload is weakly global but the pointer overload is local.

The cause of this is obscured by clang's slack diagnosing of the __attribute__ syntax extension, which after all is a GCC invention. If we compile with g++ instead we get:

$ g++ -shared -O2 -Wall -Wextra -std=c++1z -fvisibility=hidden -fPIC libtest.cpp -o libtest.so
libtest.cpp:9:36: warning: ‘visibility’ attribute ignored on non-class types [-Wattributes]
 template int * VISIBLE foobar(int *);
                                    ^

Notice that g++ ignores the visibility attribute only in the pointer overload, and, just like clang - and consistent with that warning - it emits code with:

$ nm -C libtest.so | grep foobar
0000000000000610 W int foobar<int>(int)
0000000000000620 t int* foobar<int*>(int*)

Clearly clang is doing the same thing, but not telling us why.

The difference between the overloads that satisfies g++ with one and dissatisfies it with the other is the difference between int and int *. On that basis we'd expect g++ to be satisfied with the change:

template int  VISIBLE foobar(int);
//template int * VISIBLE foobar(int *);
template float VISIBLE foobar(float);

And so it is:

$ g++ -shared -O2 -Wall -Wextra -std=c++1z -fvisibility=hidden -fPIC libtest.cpp -o libtest.so
$ nm -C libtest.so | grep foobar
0000000000000650 W float foobar<float>(float)
0000000000000640 W int foobar<int>(int)

And so is clang:

$ clang++ -shared -O2 -Wall -Wextra -std=c++1z -fvisibility=hidden -fPIC libtest.cpp -o libtest.so
$ nm -C libtest.so | grep foobar
0000000000000660 W float foobar<float>(float)
0000000000000650 W int foobar<int>(int)

Both of them will do what you want for overloads with T a non-pointer type, but not with T a pointer type.

What you face here, however, is not a ban on dynamically visible functions that return pointers rather than non-pointers. It couldn't have escaped notice if visibility was as broken as that. It is just a ban on types of the form:

D __attribute__((visibility("...")))

where D is a pointer or reference type, as distinct from types of the form:

E __attribute__((visibility("..."))) *

or:

E __attribute__((visibility("..."))) &

where E is not a pointer or reference type. The distinction is between:

• A (pointer or reference that has visibility ...) to type D

and:

• A (pointer or reference to type E) that has visibility ...

See:

$ cat demo.cpp 
int  xx ;
int __attribute__((visibility("default"))) * pvxx; // OK
int * __attribute__((visibility("default"))) vpxx; // Not OK
int __attribute__((visibility("default"))) & rvxx = xx; // OK,
int & __attribute__((visibility("default"))) vrxx = xx; // Not OK


$ g++ -shared -Wall -Wextra -std=c++1z -fvisibility=hidden -o libdemo.so demo.cpp 
demo.cpp:3:46: warning: ‘visibility’ attribute ignored on non-class types [-Wattributes]
 int * __attribute__((visibility("default"))) vpxx; // Not OK
                                              ^
demo.cpp:5:46: warning: ‘visibility’ attribute ignored on non-class types [-Wattributes]
 int & __attribute__((visibility("default"))) vrxx = xx; // Not OK
                                          ^

$ nm -C libdemo.so | grep xx
0000000000201030 B pvxx
0000000000000620 R rvxx
0000000000201038 b vpxx
0000000000000628 r vrxx
0000000000201028 b xx

The OK declarations become global symbols; the Not OK ones become local, and only the former are dynamically visible:

nm -CD libdemo.so | grep xx
0000000000201030 B pvxx
0000000000000620 R rvxx

This behaviour is reasonable. We can't expect a compiler to attribute global, dynamic visibility to a pointer or reference that could point or refer to something that does not have global or dynamic visibility.

This reasonable behaviour only appears to frustrate your objective because - as you probably now see:

template int  VISIBLE foobar(int);
template int* VISIBLE foobar(int*);

doesn't mean what you thought it did. You thought that, for given type U,

template U VISIBLE foobar(U);

declares a template instantiating function that has default visibility, accepting an argument of type U and returning the same. In fact, it declares a template instantiating function that accepts an argument of type U and returns type:

U __attribute__((visibility("default")))

which is allowed for U = int, but disallowed for U = int *.

To express your intention that instantations of template<typename T> T foobar(T arg) shall be dynamically visible functions, qualify the type of the template function itself with the visibility attribute. Per GCC's documentation of the __attribute__ syntax - which admittedly says nothing specific concerning templates - you must make an attribute qualification of a function in a declaration other than its definition. So complying with that, you'd revise your code like:

// libtest.cpp
#define VISIBLE __attribute__((visibility("default")))

template<typename T> T foobar(T arg) VISIBLE;

template<typename T> T foobar(T arg) {
    return arg;
}

template int  foobar(int);
template int* foobar(int*);

g++ no longer has any gripes:

$ g++ -shared -O2 -Wall -Wextra -std=c++1z -fvisibility=hidden -fPIC libtest.cpp -o libtest.so
$ nm -CD libtest.so | grep foobar
0000000000000640 W int foobar<int>(int)
0000000000000650 W int* foobar<int*>(int*)

and both of the overloads are dynamically visible. The same goes for clang:

$ clang++ -shared -O2 -Wall -Wextra -std=c++1z -fvisibility=hidden -fPIC libtest.cpp -o libtest.so
$ nm -CD libtest.so | grep foobar
0000000000000650 W int foobar<int>(int)
0000000000000660 W int* foobar<int*>(int*)

With any luck, you'll have the same result with clang on Mac OS