Python -- Overhead of `.so` Imports?

Question

I have 1000s of custom (compiled to '.so') modules that I'd like to use in python at the same time. Each such module is of size (100 [KB]) on average.

Does anyone know what is the overhead (on the OS -- assuming python is not handling this) of every .so import? meaning, is the overhead equal to the size of the .so file on disk? or is it a fixed, regardless of the size of the .so file?

I haven't yet gotten there, but would be curious to know what is the impact on the OS when one wants to import, say 10,000-50,000 custom modules at once.

Answer 1

There would be a large time overhead of importing that many shared libraries - the dynamic linker would spend a significant amount of time during the loading phase. The dynamic linker is really optimized for tens to hundreds of shared objects, not thousands to tens of thousands.

If at all possible, combine your shared code objects.

However, the size once loaded is likely somewhat smaller than the one disk size, depending on what other information is in the file (DWARF debug symbols, extra ELF sections not required, etc).