libpthread.so continues to use TLS space and DL namespaces after `dlclose()`

Question

I'm working on a project that needs to arbitrarily load/unload Rust-based plugins (shared objects) into isolated dynamic library namespaces.

I use dlmopen(LM_ID_NEWLM, "rust-plugin.so", RTLD_LAZY) to create a new namespace for a shared object. When the the shared object is no longer needed, I call dlclose().

Unfortunately, I've found that even when I dlclose() so that only one shared object is valid at a time, after dlmopen()ing 14 Rust plugin objects, I get the error:

dlmopen(rust-plugin.so) failed: /lib/x86_64-linux-gnu/libc.so.6: cannot allocate memory in static TLS block

Continued attempts to dlmopen() after this failure result in a segmentation fault and no more namespaces available for dlmopen().

I seem to have isolated the problem to the libpthread.so dependency of Rust shared objects. Other shared object dependencies like libgcc_s.so.1 (and any .so files I have tried, for that matter) can be opened and closed by the following code indefinitely, whereas libpthread.so errors out after I open and close it 14 times.

#include <link.h>
#include <stdio.h>
#include <dlfcn.h>

#include <cstdlib>

void load(char const *file) {
    void *handle_ = dlmopen(LM_ID_NEWLM, file, RTLD_LAZY);
    if (!handle_) {
      printf("dlmopen(%s) failed: %s\n", file, dlerror());
      exit(1);
    }
    if (dlclose(handle_) != 0) {
      exit(2);
    }
}

int main() {
  void *handle_;
  for (int i = 0; true; i++) {
    printf("%d\n", i);
    load("libpthread.so.0");
  }
}

Is there some way I can have libpthread properly clean up so I can avoid this issue?

Answer 1

libpthread.so.0 has NODELETE flag:

readelf -d /lib/x86_64-linux-gnu/libpthread.so.0 | grep NODELETE
 0x000000006ffffffb (FLAGS_1)            Flags: NODELETE INITFIRST

This makes dlclose() on it a no-op. See also this answer.

Given that dlclose() is no-op, everything else makes sense: GLIBC is configured with 16 total loader namespaces, and one of them is reserved for the main application. Once you call dlmopen (without calling dlclose) 15 times, you exhaust them all, and subsequent attempts fail with no more namespaces available.

Marking libpthread with NODELETE makes sense: once it's in the picture, it fundamentally changes GLIBC operation (e.g. malloc starts acquiring locks, errno switches to thread-local, etc. etc.).

Is there some way I can have libpthread properly clean up so I can avoid this issue?

I believe the only realistic choice for you is to try to avoid dependency on libpthread from your plugin.

Other things you could do:

1. open a bug against GLIBC (but you'll likely wait a long time for it to get fixed). I am not sure what the implications of unloading libpthread from non-default loader scope are; perhaps it should be unloadable,
2. build your own libpthread.so, identical to the system one, but without the -z,nodelete linker flag, and arrange for this libpthread.so.0 to be loaded as the plugin dependency (make sure this version stays in sync with the system one, or you'll see very hard to debug crashes).