How to figure out the reason of a SIGSEGV in Android

Question

I've been getting increasing number of reports that my android app crashes, the crash happens in native code so a stacktrace isn't produced. I have only one native function and it isn't the problem because it's called once during startup and the problem happens 2-3 minutes later.

I got the following partial dump from logcat:

I/DEBUG   ( 8704): pid: 9028, tid: 9136  >>> xxxx <<< I/DEBUG   ( 8704): at : (2012-10-30 21:10:42.000086303 UTC) 0 I/DEBUG   ( 8704): signal 11 (SIGSEGV), fault addr deadbaad I/DEBUG   ( 8704):  r0 00000000  r1 afd14691  r2 00000027  r3 00000070 I/DEBUG   ( 8704):  r4 afd42328  r5 00000000  r6 00000000  r7 0000a000 I/DEBUG   ( 8704):  r8 00000000  r9 4a77b900  10 4a77b8d8  fp 00000046 I/DEBUG   ( 8704):  ip 00001750  sp 4a77b590  lr deadbaad  pc afd11ce4  cpsr 60000030 I/DEBUG   ( 8704):  d0  643a64696f72646e  d1  6472656767756265 I/DEBUG   ( 8704):  d2  0000005d0000005b  d3  0006000600060006 I/DEBUG   ( 8704):  d4  0120009c009c009c  d5  000a000a000a000a I/DEBUG   ( 8704):  d6  0012000600060006  d7  0000000000000000 I/DEBUG   ( 8704):  d8  0000004600000000  d9  43f0000000000000 I/DEBUG   ( 8704):  d10 4040000043080000  d11 0000000043ee464c I/DEBUG   ( 8704):  d12 0000000000000000  d13 0000000000000000 I/DEBUG   ( 8704):  d14 0000000000000000  d15 0000000000000000 I/DEBUG   ( 8704):  d16 0505050507070707  d17 0b0b0b0b10101010 I/DEBUG   ( 8704):  d18 0707070703030303  d19 0000000000000000 I/DEBUG   ( 8704):  d20 0100010001000100  d21 0100010001000100 I/DEBUG   ( 8704):  d22 0000000000000000  d23 0000000000000000 I/DEBUG   ( 8704):  d24 0000000000000000  d25 0000000000000000 I/DEBUG   ( 8704):  d26 00f900f900f900f9  d27 00fb00fb00fb00fb I/DEBUG   ( 8704):  d28 00f000f000f000f0  d29 00f500f500f500f5 I/DEBUG   ( 8704):  d30 01748000016e0000  d31 01818000017b0000 I/DEBUG   ( 8704):  scr 20000012 I/DEBUG   ( 8704): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** I/DEBUG   ( 8704): Build fingerprint: \'lge/lge_bprj/lgp970/lgp970:2.2.2/FRG83G/V10a_07-JUN-2011.422D181D:user/release-keys\' I/DEBUG   ( 8704):          #00  pc 00011ce4  /system/lib/libc.so I/DEBUG   ( 8704):          #01  pc 0000be32  /system/lib/libc.so I/DEBUG   ( 8704):          #02  pc 0000cd92  /system/lib/libc.so I/DEBUG   ( 8704):          #03  pc 000008d8  /system/lib/libstdc++.so I/DEBUG   ( 8704):          #04  pc 0004efa4  /system/lib/libskia.so I/DEBUG   ( 8704):          #05  pc 0006d00c  /system/lib/libskia.so I/DEBUG   ( 8704):          #06  pc 0006f414  /system/lib/libskia.so I/DEBUG   ( 8704):  I/DEBUG   ( 8704): code around pc: I/DEBUG   ( 8704): afd11cc4 2d00682d e029d1fb b12b68db c05cf8df  I/DEBUG   ( 8704): afd11cd4 f8442001 4798000c e054f8df 26002227  I/DEBUG   ( 8704): afd11ce4 2000f88e eecaf7fb f7fc2106 f04feff8  I/DEBUG   ( 8704): afd11cf4 91035180 460aa901 96012006 f7fc9602  I/DEBUG   ( 8704): afd11d04 a905eb6e 20024632 eb78f7fc eeb6f7fb  I/DEBUG   ( 8704):  I/DEBUG   ( 8704): code around lr: I/DEBUG   ( 8704): deadba8c ffffffff ffffffff ffffffff ffffffff  I/DEBUG   ( 8704): deadba9c ffffffff ffffffff ffffffff ffffffff  I/DEBUG   ( 8704): deadbaac ffffffff ffffffff ffffffff ffffffff  I/DEBUG   ( 8704): deadbabc ffffffff ffffffff ffffffff ffffffff  I/DEBUG   ( 8704): deadbacc ffffffff ffffffff ffffffff ffffffff  I/DEBUG   ( 8704):  I/DEBUG   ( 8704): stack: I/DEBUG   ( 8704):     4a77b550  00000015   I/DEBUG   ( 8704):     4a77b554  afd146c1  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b558  afd425a0  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b55c  afd4254c  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b560  00000000   I/DEBUG   ( 8704):     4a77b564  afd156db  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b568  afd14691  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b56c  afd14691  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b570  00000070   I/DEBUG   ( 8704):     4a77b574  afd42328  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b578  00000000   I/DEBUG   ( 8704):     4a77b57c  4a77b5a4   I/DEBUG   ( 8704):     4a77b580  0000a000  [heap] I/DEBUG   ( 8704):     4a77b584  afd14933  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b588  df002777   I/DEBUG   ( 8704):     4a77b58c  e3a070ad   I/DEBUG   ( 8704): #00 4a77b590  afd43904  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b594  afd103ec  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b598  afd42328  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b59c  49687628   I/DEBUG   ( 8704):     4a77b5a0  49687628   I/DEBUG   ( 8704):     4a77b5a4  fffffbdf   I/DEBUG   ( 8704):     4a77b5a8  afd42328  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b5ac  afd4374c  /system/lib/libc.so I/DEBUG   ( 8704):     4a77b5b0  49687628   I/DEBUG   ( 8704):     4a77b5b4  afd0be37  /system/lib/libc.so I/DEBUG   ( 8704): #01 4a77b5b8  49687660   I/DEBUG   ( 8704):     4a77b5bc  00430a60  [heap] I/DEBUG   ( 8704):     4a77b5c0  00001424   I/DEBUG   ( 8704):     4a77b5c4  4a77b8b0   I/DEBUG   ( 8704):     4a77b5c8  00000000   I/DEBUG   ( 8704):     4a77b5cc  49687628   I/DEBUG   ( 8704):     4a77b5d0  49687628   I/DEBUG   ( 8704):     4a77b5d4  00000000   I/DEBUG   ( 8704):     4a77b5d8  4a77b8b0   I/DEBUG   ( 8704):     4a77b5dc  afd0cd95  /system/lib/libc.so 

My question:

I know that I can find the native function in which the sigsegv happens from this, but I'm not sure where to get the native library to find this in. It happens on the phone of a user, I have no direct access to it or to the user... Is there a website with these libraries or can I build them while they stay the same as the one on the phone ? Or is there a better way to debug these ?

Thanks for reading, any idea is welcome :-)

Update:

As suggested, using ndk-stack like so:

./ndk-stack -sym LONGPATH/armeabi -dump foo.txt 

where foo.txt contains the "Build fingerprint" stuff, starting with a line with lots of stars (it searches for that), but all I get is:

********** Crash dump: ********** Build fingerprint: \'lge/lge_bprj/lgp970/lgp970:2.2.2/FRG83G/V10a_07-JUN-2011.422D181D:user/release-keys\' 

and the program stops..

edit2:

Using arm-linux-androideabi-addr2line and a downloaded FRG83G rom I got a little further:

it points to alarm, dlmalloc_trim and realloc in libc.so, to delete in ligstdc++.so and to SI8_D16_filter_DXDY, drawBitmap, clipQuad in libskia.so

Now comes the fun part, what to do with this information.. I'll continue tomorrow, it's 0530 here..

edit3:

I'm one step further but stuck again, the pc values in the dump point to functions in libskia.so, I found that using a Paint object in multiple threads at the same time can cause this problem, I don't do that but I did re-use BitmapFactory.Options, hoping that that was the problem, I removed that part, still crashing... The dump points to SkBitmapProcShader::SkBitmapProcShader, SkTriColorShader::shadeSpan and SkDraw::drawBitmap. The last one is the most obvious for pointing to bitmap problems.. But it's not BitmapFactory.Options or Paint objects..

Answer 1

"I have only one native function and it isn't the problem because it's called once during startup and the problem happens 2-3 minutes later."

Bzzzztt! Wrong. It's entirely possible to trash memory in one place .... then fail when you try to access the (now corrupt) memory in a completely different part of the program.

I assume you're using the NDK.

Suggestions:

1) put "sentinel" values around any global memory or malloc'ed buffer.

2) Carefully double-check all your "malloc()'s" ("deadbaad" sounds suspiciously like something that's been allocated from the heap)

3) Look here for tips on using ndk-gdb:

• http://mhandroid.wordpress.com/2011/01/25/how-cc-debugging-works-on-android/

PS:

Is your native code C or C++?