dlclose doesn't really unload shared object, no matter how many times it is called

Question

My program uses dlopen to load a shared object and later dlclose to unload it. Sometimes this shared object is loaded once again. I noticed static variables are not re-initialized (something which is crucial to my program) so I added a test (dlopen with RTLD_NOLOAD) after dlclose to see if the library is really unloaded. Sure enough, it was still in memory.

I then tried calling dlclose repeatedly until the library is really unloaded, but what I got was an infinite loop. This is the code I'm using to check if the library was unloaded:

dlclose(handles[name]);

do {
  void *handle = dlopen(filenames[name], RTLD_NOW | RTLD_NOLOAD);
  if (!handle)
    break;

  dlclose(handle);
} while (true);

My question is, what are the possible reasons for my shared object not being unloaded after dlclose, given that my dlopen calls are the only places where it is loaded. Can you suggest a course of action to track down the source of the problem? Also, why are repeated calls to dlclose have no effect, they are each decrementing the reference count, aren't they?

EDIT: Just found out that this happens only when I compile with gcc. With clang, everything is just fine.

Answer 1

The POSIX standard actually does not require dlclose to ever unload a library from address space:

Although a dlclose() operation is not required to remove structures from an address space, neither is an implementation prohibited from doing so.

Source: The Open Group Base Specifications Issue 6

That means other than invalidating the handle, dlclose is not required to do anything at all.

Sometimes unloading is also delayed by the system, it just marks the library as "to be removed" and will actually perform that operation at some later time (for efficiency or because it would simply not be possible to perform that operation right now). However, if you call dlopen again before it ever was performed, the flag is cleared and the still loaded library is reused.

In some cases the system knows for sure that some symbols of the library are still in use, in that case it will not unload it from address space to avoid dangling pointers. In some cases the system doesn't know for sure that they are in use, but it also can impossibly tell for sure that they are not, better being safe than sorry, it will just never really remove that library from memory in such a case.

There are other more obscure cases depending on operation system kind and often also on version. E.g. a common Linux issue is if you created a library that uses STB_GNU_UNIQUE symbols, that library is marked as "not unloadable" and thus will simply never be unloaded. See here, here (DF_1_NODELETE means not unloadable) and here. So it can also depend on what symbols or kind of symbol a compiler generates. Try running readelf -Ws on your library and look for objects tagged as UNIQUE.

In general, you cannot really rely on dlclose to work as you might expect. In practice I saw it "fail" more often than "succeed" in the last ten years (well, it never really failed, it just often did not unload the library from memory; yet it worked as required by the standards).

Answer 2

This is not the answer to all your questions, but this is the solution that can help you avoid problems with dlclose. This question suggests a clue about how to affect behaviour of re-loading shared libraries: you may use compiler flag -fno-gnu-unique.

From man pages for gcc / g++:

-fno-gnu-unique

On systems with recent GNU assembler and C library, the C++ compiler uses the "STB_GNU_UNIQUE" binding to make sure that definitions of template static data members and static local variables in inline functions are unique even in the presence of "RTLD_LOCAL"; this is necessary to avoid problems with a library used by two different "RTLD_LOCAL" plugins depending on a definition in one of them and therefore disagreeing with the other one about the binding of the symbol. But this causes "dlclose" to be ignored for affected DSOs; if your program relies on reinitialization of a DSO via "dlclose" and "dlopen", you can use -fno-gnu-unique.

Whether -fno-gnu-unique is used by default or not depends on how GCC has been configured: --disable-gnu-unique-object enables this flag by default, --enable-gnu-unique-object disables it.