checking shared libraries for non default loaders

Question

ldd is a good simple way to check for shared libraries a given executable is or will be using. However it does not always work as expected. For example, see the following shell snippet that demonstrates how it "fails" to found the libreadline "dependency" into the python binary

I've tried many other distributions, but I'm copying from Tikanga

$ lsb_release -a
LSB Version:    :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 5.6 (Tikanga)
Release:        5.6
Codename:       Tikanga

See what ldd does on the default installed python (from official repositories).

$ which python
/usr/bin/python
$ ldd `which python`
    libpython2.4.so.1.0 => /usr/lib64/libpython2.4.so.1.0 (0x00000030e6200000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00000030e0e00000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00000030e0a00000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00000030ee800000)
    libm.so.6 => /lib64/libm.so.6 (0x00000030e0600000)
    libc.so.6 => /lib64/libc.so.6 (0x00000030e0200000)
    /lib64/ld-linux-x86-64.so.2 (0x00000030dfe00000)
$ ldd `which python` | grep readline
$

Nothing found about readline. Now I do know from interactive usage that this binary does have realine functionality, so let't try to see where it's coming from.

$ python &
[1] 21003
$ Python 2.4.3 (#1, Dec 10 2010, 17:24:35) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-50)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

[1]+  Stopped                 python

Started an interactive python session in background (pid 21003)

$ lsof -p 21003
COMMAND   PID    USER   FD   TYPE DEVICE     SIZE    NODE NAME
python  21003 ddvento  cwd    DIR   0,33    16384  164304 /glade/home/ddvento/loader-test
python  21003 ddvento  rtd    DIR    8,3     4096       2 /
python  21003 ddvento  txt    REG    8,3     8304 6813419 /usr/bin/python
python  21003 ddvento  mem    REG    8,3   143600 8699326 /lib64/ld-2.5.so
python  21003 ddvento  mem    REG    8,3  1722304 8699327 /lib64/libc-2.5.so
python  21003 ddvento  mem    REG    8,3   615136 8699490 /lib64/libm-2.5.so
python  21003 ddvento  mem    REG    8,3    23360 8699458 /lib64/libdl-2.5.so
python  21003 ddvento  mem    REG    8,3   145824 8699445 /lib64/libpthread-2.5.so
python  21003 ddvento  mem    REG    8,3   247544 6821551 /usr/lib64/libreadline.so.5.1
python  21003 ddvento  mem    REG    8,3    15840 8699446 /lib64/libtermcap.so.2.0.8
python  21003 ddvento  mem    REG    8,3  1244792 6833317 /usr/lib64/libpython2.4.so.1.0
python  21003 ddvento  mem    REG    8,3    18152 8699626 /lib64/libutil-2.5.so
python  21003 ddvento  mem    REG    8,3 56446448 6832889 /usr/lib/locale/locale-archive
python  21003 ddvento  mem    REG    8,3    21808 6965997 /usr/lib64/python2.4/lib-dynload/readline.so
python  21003 ddvento  mem    REG    8,3    25464 6901074 /usr/lib64/gconv/gconv-modules.cache
python  21003 ddvento    0u   CHR  136,1                3 /dev/pts/1
python  21003 ddvento    1u   CHR  136,1                3 /dev/pts/1
python  21003 ddvento    2u   CHR  136,1                3 /dev/pts/1
$ lsof -p 21003 | grep readline
python  21003 ddvento  mem    REG    8,3   247544 6821551 /usr/lib64/libreadline.so.5.1
python  21003 ddvento  mem    REG    8,3    21808 6965997 /usr/lib64/python2.4/lib-dynload/readline.so

Bingo! Here it is readline!

However, this technique works only when the library is effectively loaded, so for example it does not find /usr/lib64/libtcl8.4.so until the python process does not run something like from Tkinter import *

So I have two questions:

1. I believe the problem with ldd is that it assumes the usage of the standard loader, whereas very likely python is using its own special loader (so that you don't have to relink the executable every time you install a new python module that is not pure python but has some c/c++/fortran code). Is this correct?

2. Clearly, if an executable is using its own loader, there is no obvious answer to the question "how to find all the possible libraries this executable may load": it depends on what the loader does. But is there a way to find out what libraries may be loaded by python?

PS: related to 1. If you are landing on this question you should already know the following, but if don't you should: see how simple is to completely mess up ldd output (messing it up only partially is a little harder):

$ cat hello.c 
#include <stdio.h>

int main() {
  printf("Hello world.\n");
  return 0;
}

$ gcc -static hello.c -o loader
$ gcc -Wl,--dynamic-linker,./loader hello.c -o hello
$ ./hello 
Hello world.
$ ldd ./hello
Hello world.

Answer 1

Python, Perl, and other interpreted languages do load things dynamically using dlopen(). (This is not the same thing as replacing the standard loader; they are still using that, and in fact dlopen() is a hook into the standard loader on ELF-based systems.)

There is no standard registry for loadable modules. Python uses its own rules to determine where extension modules can be loaded from (look at sys.path), including those which have associated shared objects. Perl uses different rules. Apache uses still different rules, etc.

So to summarize the answers to your questions:

1. not exactly

2. no

Answer 2

As a side note, a possible way to accomplish what I wanted in question 2 would be to:

• create an empty chrooted environment

• recompile python in there, manually adding anything that is missing, one-by-one

Depending on your goals, this might or might not be a good solution (and actually turns out to be not too bad for what my goals are - strange as it may sound from the question)