atos and dwarfdump won't symbolicate my address

Question

I received a crash report via AirBrake.io that isn't symbolicated. Since the crash report is not in exactly the same format as an Apple crashlog I can't just drop it on XCode as usual, so I took the exact same build from my XCode archive tried to symbolicate it on the commandline. With the following result:

$ atos -o kidsapp.app/kidsapp 0x0002fc4c
0x0002fc4c (in kidsapp)

I'm absolutely sure I'm using the same build as the crash report is from. So I also tried with dwarfdump:

$ dwarfdump --lookup 0x0002fc4c --arch armv7 kidsapp.app.dSYM
----------------------------------------------------------------------
 File: kidsapp.app.dSYM/Contents/Resources/DWARF/kidsapp (armv7)
----------------------------------------------------------------------
Looking up address: 0x000000000002fc4c in .debug_info... not found.
Looking up address: 0x000000000002fc4c in .debug_frame... not found.

Also no result. Is there anything else besides using the wrong dSYM file that I could do wrong? I know it's the correct one since this is the version referred in the crash report in AirBrake and it's in my XCode archive.

Any ideas/tips are welcome!

Answer 1

I used the following arithmetic to figure it out:

slide + stack address - load address = symbol address

and

stack address is the hex value I get from my stack dump crash report (not a .crash file, just the stack dump).

and

slide is the vmaddr of the LC_SEGMENT cmd when running otool -arch armv7 -l APP_BINARY_PATH. Mine usually ends up being 0x00001000.

and

load address is the complicated piece. It is actually the difference between the bottommost stack address of the main thread and the FIRST address of the portion of my binary that contains symbols when running dwarfdump --arch armv7 --all DSYM_BINARY_PATH. This is simply the symbolic address of the main function. So if your bottom most crash address is 0x8000 and your main function's symbolic address is 0x2000 then your load address is 0x6000.

Now with ALL these pieces I can calculate the symbol address and put that into atos or dwarfdump: dwarfdump --lookup SYM_ADDR --arch armv7 APP_BINARY_PATH.

Example of the dump (you can see that the load address was 0x00003af4):

----------------------------------------------------------------------

File: /Users/user/Desktop/MyApp.xcarchive/dSYMs/MyApp.app.dSYM/Contents/Resources/DWARF/MyApp (armv7)

----------------------------------------------------------------------

0x00000024: [0x00003af4 - 0x00003b4e) main

0x00000098: [0x00003b50 - 0x00003d8c) -[MyAppDelegate application: didFinishLaunchingWithOptions:]

... the rest of the dump

The hardest part was realizing that one of the 2 static libraries I'd included had their symbols stripped before being link to my app's binary! That left a HUGE gap of symbol addresses so I only ended up with two-thirds of the symbols I needed in my dSYM.

Be sure to have the following flags set to NO in your static libraries xcode project so that when you link against it, you can pull in the symbols to your app's binary (which can later be stripped): COPY_PHASE_STRIP, DEAD_CODE_STRIPPING, and STRIP_INSTALLED_PRODUCT.

Now you may ask, "what do I do if the stack dump does not include the main function since it isn't on the main thread so that I cannot get the main function's stack address?". To that I would reply, "I haven't a friggin' clue!". Just cross your fingers and hope you can get a stack trace that includes the symbol address or use a crash reporting system that mimics Apple's crash logs, like PLCrashReporter.

[EDIT May 26, 2013] -

It was brought to my attention that the load address is really the address of the mach-o binary. Though what I described above can often work - it's not actually correct. This can be obtained via the CRASH REPORT, however the point of this answer was to provide the symbols of a crash when you don't have a crash report. The best way I've come to figuring out the load address when wanting to symbolicate is by making sure I log the load address with the stack addresses.

I've personally created a system for logging crashes (not crash reports) and having them sent to an S3 bucket where I can retrieve them later for debugging. When I start my application I cache the slide, the load address and the main function address for use if my app crashes and I send up a the stack addresses.

NOTE: the dyld functions use #include <mach-o/dyld.h>

slide = the address returned by _dyld_get_image_vmaddr_slide(0)

load address = the address returned by _dyld_get_image_header(0)

main function address = the last address in [NSThread callStackReturnAddresses] when called on the main thread

At crash time I'm sure to log [NSThread callStackReturnAddresses] and [NSThread callStackSymbols] as well as the architecture which can be retrieve by having this method:

- (NSString*) arch
{
    NSString* arch =
#ifdef _ARM_ARCH_7
        @"armv7";
#elif defined (_ARM_ARCH_6)
        @"armv6";
#else
        nil;
#endif

    return arch;
}

I don't yet know how to differentiate between armv7 and armv7s though.

So this may help in the future. I plan on taking everything I've learned and turning this into a simple crash tool - better than the natos tool (probably natos v2).

I've updated natos to support supplying the load address manually: https://github.com/NSProgrammer/natos