How to debug program with custom elf interpreter?

Question

I can debug some program (say /bin/ls) like this:

[ks@localhost ~]$ gdb -q --args /bin/ls
Reading symbols from /bin/ls...Reading symbols from /bin/ls...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install coreutils-8.22-19.fc21.x86_64
(gdb) start 
Temporary breakpoint 1 at 0x402990
Starting program: /usr/bin/ls 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Temporary breakpoint 1, 0x0000000000402990 in main ()
(gdb) 

Here I can set temporary breakpoint at main and stop at it.

But I have to run program with custom elf interpreter like this:

[ks@localhost ~]$ gdb -q --args /lib64/ld-linux-x86-64.so.2 /bin/ls
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.20.so.debug...done.
done.
(gdb) start 
Function "main" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Temporary breakpoint 1 (main) pending.
Starting program: /usr/lib64/ld-linux-x86-64.so.2 /bin/ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
1234            glibc-2.20      python     tmp
[Inferior 1 (process 2610) exited normally]
Missing separate debuginfos, use: debuginfo-install libacl-2.2.52-7.fc21.x86_64 libattr-2.4.47-9.fc21.x86_64 libcap-2.24-7.fc21.x86_64 pcre-8.35-8.fc21.x86_64
(gdb) 

Here gdb did not stop at main because symbols for /bin/ls were not loaded. How can I force gdb to load symbols and stop at main in this case?

Answer 1

Here is how you can do it:

cat t.c
#include <stdio.h>
#include <stdlib.h>

int main()
{
  printf("Hello\n");
  return 0;
}

gcc -g t.c
gdb -q --args /usr/lib64/ld-linux-x86-64.so.2 ./a.out
(gdb) start
Function "main" not defined.
Starting program: /usr/lib64/ld-linux-x86-64.so.2 ./a.out
Hello
[Inferior 1 (process 7134) exited normally]

So far everything is matching what you observed. Now for the solution:

(gdb) set stop-on-solib-events 1
(gdb) r
Starting program: /usr/lib64/ld-linux-x86-64.so.2 ./a.out
Stopped due to shared library event (no libraries added or removed)
(gdb) c
Continuing.
Stopped due to shared library event:
  Inferior loaded /usr/lib64/ld-linux-x86-64.so.2
(gdb) c
Continuing.
Stopped due to shared library event:
  Inferior loaded /usr/lib64/libc.so.6

At this point, ./a.out has also been loaded, and you can confirm that with:

(gdb) info proc map
process 7140
Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
            0x400000           0x401000     0x1000        0x0 /tmp/a.out
            0x600000           0x601000     0x1000        0x0 /tmp/a.out
            0x601000           0x602000     0x1000     0x1000 /tmp/a.out
      0x555555554000     0x555555579000    0x25000        0x0 /usr/lib64/ld-2.19.so
      0x555555779000     0x55555577a000     0x1000    0x25000 /usr/lib64/ld-2.19.so
      0x55555577a000     0x55555577c000     0x2000    0x26000 /usr/lib64/ld-2.19.so
      0x7ffff7c2a000     0x7ffff7c2d000     0x3000        0x0
      0x7ffff7c2d000     0x7ffff7df0000   0x1c3000        0x0 /usr/lib64/libc-2.19.so
      0x7ffff7df0000     0x7ffff7fef000   0x1ff000   0x1c3000 /usr/lib64/libc-2.19.so
      0x7ffff7fef000     0x7ffff7ff3000     0x4000   0x1c2000 /usr/lib64/libc-2.19.so
      0x7ffff7ff3000     0x7ffff7ff5000     0x2000   0x1c6000 /usr/lib64/libc-2.19.so
      0x7ffff7ff5000     0x7ffff7ff9000     0x4000        0x0
      0x7ffff7ff9000     0x7ffff7ffa000     0x1000        0x0 /etc/ld.so.cache
      0x7ffff7ffa000     0x7ffff7ffd000     0x3000        0x0
      0x7ffff7ffd000     0x7ffff7fff000     0x2000        0x0 [vdso]
      0x7ffffffde000     0x7ffffffff000    0x21000        0x0 [stack]
  0xffffffffff600000 0xffffffffff601000     0x1000        0x0 [vsyscall]

Unfortunately, GDB does not understand that it should also load symbols for ./a.out. You have to tell it:

(gdb) add-symbol-file ./a.out
The address where ./a.out has been loaded is missing

One would think that the address that GDB needs would be from the above info proc map: 0x400000. One would be wrong. The actual address GDB needs is the start of .text section, which you can get from readelf:

readelf -WS ./a.out | grep text
   [13] .text             PROGBITS        0000000000400440 000440 000182 00  AX  0   0 16

Back to GDB:

(gdb) add-symbol-file ./a.out 0x0000000000400440
add symbol table from file "./a.out" at
    .text_addr = 0x400440
Reading symbols from ./a.out...done.

And now we can break on main:

(gdb) b main
Breakpoint 1 at 0x400531: file t.c, line 6.
(gdb) c
Continuing.

Breakpoint 1, main () at t.c:6
6     printf("Hello\n");
(gdb) n
Hello
7     return 0;

Voila!

P.S. Re-running the binary may give you some glitches:

(gdb) r
Starting program: /usr/lib64/ld-linux-x86-64.so.2 ./a.out
Error in re-setting breakpoint 1: Cannot access memory at address 0x40052d
Error in re-setting breakpoint 1: Cannot access memory at address 0x40052d
Stopped due to shared library event (no libraries added or removed)

This is happening because the ld-linux is yet to map the ./a.out. But you can continue:

(gdb) c
Continuing.
Stopped due to shared library event:
  Inferior loaded /usr/lib64/ld-linux-x86-64.so.2
(gdb) c
Continuing.
Stopped due to shared library event:
  Inferior loaded /usr/lib64/libc.so.6

And now, ./a.out has also been loaded, so you can re-enable the breakpoint(s):

(gdb) enable
(gdb) continue
Continuing.

Breakpoint 1, main () at t.c:6
6     printf("Hello\n");