dlclose() not unloading .so-file which is linking to boost

Question

If my app loads (using dlopen) a .so file, that is linking to the Boost Test Framework, I can't unload the so file. Without linking to boost it seems to be fine to unload it.

App file main.cpp:

#include <dlfcn.h>
#include <iostream>

int main()
{
   auto sFileName = "./libtest_library.so";
   auto handle = dlopen(sFileName, RTLD_LAZY | RTLD_LOCAL);

   if (!handle) 
      std::cerr << "Error: " << dlerror() << std::endl;

   auto closing = dlclose(handle);
   while(1);
   return 0;
}

Library .so file (libtest_library.so):

#include <iostream>
//#include "boost/test/unit_test.hpp"

static void con() __attribute__((constructor));
static void dcon() __attribute__((destructor));

void con()
{
   std::cout << "Constructing library..." << std::endl;
}

void dcon()
{
   std::cout << "Destructing library..." << std::endl;
}

Running this I get the output:

Constructing library...
Destructing library...

If I link to Boost's unit test framework in libtest_library.so I only get the Constructing library... output. dlclose(handle) returns 0 (which is success).

Currently linking against Boost v. 1.60.0, compiling with gcc 5_2_0 on Ubuntu 14.04. Is this a bug in Boost? Compiler? Any ideas?

I need to reload the .so files several times in a project, and it needs to be fully unloaded (not existing in the memory). How can I solve this? Thanks.

---

Update 1:
It seems like if I only link to boost the libtest_library destructor is actually called but the boost_test_framework library is not unloaded. However, if I include "boost/test/unit_test.hpp", the destructor won't be called (libtest_library.so refuse to unload).

Update 2:
Looking through boost's sources I’ve found that there is a c++ singleton in boost causing the problem.

I can replicate the problem in a simplified version. Basically if I add the following singleton to the libtest_library it doesn’t work (can’t unload the .so file):

alt 1

class Singleton
{
public:
   static Singleton & getInstance() { static Singleton instance; return instance; }    
private:
   Singleton() {}
   ~Singleton() {}
};

static Singleton & singleton = Singleton::getInstance();

But using this works:

alt 2

class Singleton
{
public:
    static Singleton & getInstance();
private:
   Singleton() {}
   ~Singleton() {}
};

Singleton & Singleton::getInstance() { static Singleton instance; return instance; }

static Singleton & singleton = Singleton::getInstance();

I’ve tried the different GCC compilers and it leads to the same results. For me this seems to be bug?

Also the symbols are a bit different: doing nm –C libtest_library.so | grep –i singleton I get

alt 1 (not working one):

0000000000201460 u guard variable for Singleton::getInstance()::instance
0000000000201458 b singleton
0000000000000e66 W Singleton::getInstance()
0000000000000f08 W Singleton::Singleton()
0000000000000f08 W Singleton::Singleton()
0000000000000f1c W Singleton::~Singleton()
0000000000000f1c W Singleton::~Singleton()
0000000000201468 u Singleton::getInstance()::instance

And alt 2:

00000000002012f8 b guard variable for Singleton::getInstance()::instance
0000000000201300 b singleton
0000000000000bb0 T Singleton::getInstance()
0000000000000cec W Singleton::Singleton()
0000000000000cec W Singleton::Singleton()
0000000000000d00 W Singleton::~Singleton()
0000000000000d00 W Singleton::~Singleton()
0000000000201308 b Singleton::getInstance()::instance

Any Ideas?

Update 3

I've extracted out the part in boost that seems to create the problem and created a minimal example that demonstrate the problem:

main_app.cpp - main app

#include <dlfcn.h>
#include <iostream>

int main()
{
   for(auto i = 0; i < 2; i++) {
      auto sFileName = "./libtest_library.so";
      auto handle = dlopen(sFileName, RTLD_LAZY | RTLD_LOCAL);

      if (!handle) {
         printf("Dlerror: %s\n", dlerror());
         continue;
      }

      auto closing = dlclose(handle);
      printf("Dlerror: %s\n", dlerror());
   }

   return 0;
}

main_lib.cpp - libtest_library.so

#include <iostream>

template<typename Derived>
class trivial_singleton_t {
public:
   static Derived& instance() { static Derived the_inst; return the_inst; }
protected:
   trivial_singleton_t() {}
   ~trivial_singleton_t() {}
};

class singleton_t : public trivial_singleton_t<singleton_t> {

private:
   friend class trivial_singleton_t<singleton_t>;
   singleton_t() {}
};

singleton_t & singleton = singleton_t::instance();

static void con() __attribute__((constructor));
static void dcon() __attribute__((destructor));

void con()
{
   std::cout << "Constructing library..." << std::endl;
}

void dcon()
{
   std::cout << "Destructing library..." << std::endl;
}

I get the following output:

Constructing library...
Dlerror: (null)
DLerror: (null)
Destructing library...

hence the library is only unloaded when main exists.

Answer 1

As noted in the question there are STB_GNU_UNIQUE symbols in the compiled binary files.

The problem is that a library that is using those symbols and loaded with dlopen will be flagged as NODELETE, and therefore persist between the dlopen/dlclose calls. See line 445 here: http://osxr.org:8080/glibc/source/elf/dl-lookup.c

https://sourceware.org/binutils/docs/binutils/nm.html, STB_GNU_UNIQUE or u:

The symbol is a unique global symbol. This is a GNU extension to the standard set of ELF symbol bindings. For such a symbol the dynamic linker will make sure that in the entire process there is just one symbol with this name and type in use.

These are made when variables/methods are created within an anonymous namespace or as static global variables.

The quickest solution is to force the compiler to not build those symbols as STB_GNU_UNIQUE using a linker flag --no-gnu-unique.

Unfortunately this didn't work for me since I didn't have a sufficiently recent linker, luckily I could rebuild gcc with the following configuration option: --disable-gnu-unique-object. Remember to also rebuild the boost library using the linker flag or new gcc.