gdb: Cannot find new threads: generic error after system update

Question

I am running an OpenEmbedded based Linux on an ARM board, where my application is running. I used to run kernel 2.6.35, gdb 6.8 and gcc 4.3. Lately I've updated the system to kernel 2.6.37, gdb 7.4 (also tried 7.3) and gcc 4.6.

Now, my application can not be debugged anymore (on the ARM board), everytime I try to run it in gdb I get the error "gdb: Cannot find new threads: generic error". The application makes use of pthreads and does link against pthreads (readelf lists libpthread.so.0 as a dependency). The suggested solutions I found so far all recommend linking to pthread which I am already doing. The other recommendation I found was to use LD_PRELOAD=/lib/libpthread.so.0 which does not make any difference for me.

Debugging the x86 builds of the application works without a problem.

EDIT: To answer the questions posed in the first answer, I am using gdb on the target (ARM), i.e. no cross-gdb. I also have not stripped libpthread.so.0 (/lib/libpthread-2.9.so: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, not stripped). glibc remained at version 2.9, and the update involved recompiling the whole linux image

EDIT2: Removing /lib/libthread-db* allows debugging (with consequent warnings and obviously some features will not work)

EDIT3: Using set debug libthread-db 1 I get:

Starting program: /home/root/app
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1.
td_ta_new failed: application not linked with libthread
thread_db_load_search returning 0
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
warning: Unable to set global thread event mask: generic error
Warning: find_new_threads_once: find_new_threads_callback: cannot get thread info: generic error
Found 0 new threads in iteration 0.
Warning: find_new_threads_once: find_new_threads_callback: cannot get thread info: generic error
Found 0 new threads in iteration 1.
Warning: find_new_threads_once: find_new_threads_callback: cannot get thread info: generic error
Found 0 new threads in iteration 2.
Warning: find_new_threads_once: find_new_threads_callback: cannot get thread info: generic error
Found 0 new threads in iteration 3.
thread_db_load_search returning 1
Warning: find_new_threads_once: find_new_threads_callback: cannot get thread info: generic error
Found 0 new threads in iteration 0.
Cannot find new threads: generic error
(gdb) Write failed: Broken pipe

Answer 1

There are two common causes of this error:

1. You have a mis-match between libpthread.so.0 and libthread_db.so.1
2. You have stripped libpthread.so.0

Your message isn't entirely clear:

1. are you using cross GDB to debug the application running on ARM from an x86 host?
2. have you updated (or rebuilt) glibc in addition to updating the kernel, etc.

If you stripped libpthread.so.0, then don't do that -- libthread_db needs it to not be stripped.

If you are cross-debugging, make sure to rebuild libthread_db.so.1 on host to match glibc on target.

Update:

not cross-debugging
did not strip libpthread

So, something in your GDB or glibc appears to have been broken. You can try to see what that is by

1. Putting removed libthread_db back, and
2. (gdb) set debug libthread-db 1
3. (gdb) run

Update 2:

warning: Unable to set global thread event mask: generic error

This means that GDB was able to look up td_ta_set_event function in libthread_db, and called it, but the function returned an error. One way this could happen is if GDB was unable to find __nptl_threads_events function in libpthread.so.0. What does this command produce:

nm /lib/libpthread.so.0 | grep __nptl_threads_events

If that command produces output, e.g.:

000000000021c294 b __nptl_threads_events

then I am not sure what else is failing. You'll likely have to debug GDB itself to figure out what's happening.

If on the other hand the grep above produces no output, then it's a problem with your toolchain: you'll have to figure out why that variable doesn't appear in your rebuilt libpthread.so.0.