stack traces stop before getting to my code (on Android using NDK)

Question

I'm developing on Android 2.3.x using NDK r5b. Occasionally my code crashes and I'd like to know where. I already know how to get the corresponding line in my application when I have a have a pointer (i.e. from Android's stack traces.)

However, oftentimes I see useless stack traces like this (full stack trace):

     #00  pc 0006561a  /system/lib/egl/libGLESv2_adreno200.so
     #01  pc 0006b900  /system/lib/egl/libGLESv2_adreno200.so
     #02  pc 0005aac8  /system/lib/egl/libGLESv2_adreno200.so
     #03  pc 0001687a  /system/lib/egl/libGLESv1_CM_adreno200.so
     #04  pc 000096ce  /system/lib/egl/libGLESv1_CM_adreno200.so

or this:

(gdb) bt
#0  0xafd0c51c in epoll_wait () from /Volumes/SecureCode/webos/rta/android/obj/local/armeabi/libc.so
#1  0xa81216a6 in ?? ()

that don't even mention my code at all.

Is there any way at all to get better stack traces than this? Why are some library functions "opaque" in that they don't allow the backtrace to "see through" to the calling function, causing a stop in the stack trace?

As far as I can tell, the only way to debug a problem like this is to use logging at each point in the program and/or step through each line with gdb.

Are there ROMs available with debug versions of these Android libraries instead of runtime ones, and would that help? (I use one phone solely for development, so I'm not concerned about keeping full functionality.) (Actually, I noticed that the path to libc.so in the above gdb stack trace is within my application directory. Could I possibly just package it with a different (debug) libc.so, and would that help?)

One last thing that might help: in the above stack trace from logcat (the first one,) my library is mentioned in the raw stack dump:

stack:
  ...
  ...
  4471cb88  00000028  
  4471cb8c  afd4649c  
  4471cb90  80b4eb71  /data/data/com.audia.dev.rta/lib/librta.so
  4471cb94  00299180  
  ...
  ...

but that is not a function pointer. What could that be, and would it be of any help after the app has crashed? I'm guessing probably not if it's a heap pointer or something like that.

Answer 1

Is there any way at all to get better stack traces than this?

As far as I know, you must build and write the Android image by yourself. It enables you to have the whole complete symbols of the Android (executable files and shared libraries) except proprietary shared libraries.

• Building from source - Compile CyanogenMod from Source

Also it provides to use the symbols using gdb.

$ adb shell setprop debug.db.uid 32767
$ adb forward tcp:5039 tcp:5039

/*
 program terminated and debuggerd caught exception like the following.
 Use the PID number for gdbclient 3rd parameter.
 I/DEBUG   ( 2154): ******************************************************** 
 I/DEBUG   ( 2154): * Process 2508 has been suspended while crashing.  To
 I/DEBUG   ( 2154): * attach gdbserver for a gdb connection on port 5039:
 I/DEBUG   ( 2154): *
 I/DEBUG   ( 2154): *     adb shell gdbserver :5039 --attach 2508 &
 I/DEBUG   ( 2154): *
 I/DEBUG   ( 2154): * Press HOME key to let the process continue crashing.
 I/DEBUG   ( 2154): ********************************************************)
*/

$ gdbclient "" "" 2508

EDITED:

You can still use ndk-gdb instead of gdbclient command. Please specify the symbol files for shared libraries.

(gdb) set solib-search-path (ANDROID_SOURCE_PATH)/out/target/product/(PRODUCT_NAME)/symbols/system/lib

EDITED 2:

If you don't need the symbols of the Android system shared libraries, just adb pull shared libraries and set sollib-search-path to it.

$ adb pull /system/lib lib

$ ndk-gdb
...
(gdb) set solib-search-path lib

Answer 2

Couple of notes:

• In some cases, your stack trace may be damaged because your stack has been partially trashed. Unlikely though.
• What OS are you using? Gingerbread (Android 2.3) is much better in terms of stack traces. If you're not running Android 2.3, find an Android 2.3 ROM for your phone somewhere, or get a cheap development phone that runs 2.3.
• Have you seen Onur's script? It's been working very well for me, even on Android 2.2 phones.
• Hope that fadden is reading this, I'm sure he has an answer that is much more helpful than mine.