OS X kernel panic diagnostics. How to translate backtrace addresses

Question

I'm debugging a driver that cause kernel dump on my mac. It shows long backtrace composed out of series of addresses:

panic(cpu 6 caller 0xffffff8004dc9986): trying to interlock destroyed mutex  
(0xffffff8049deedb0)
Backtrace (CPU 6), Frame : Return Address
0xffffff93b1c8bb50 : 0xffffff8004ce5307 
0xffffff93b1c8bbd0 : 0xffffff8004dc9986 
0xffffff93b1c8bbe0 : 0xffffff8004d099eb 
0xffffff93b1c8bc20 : 0xffffff7f85604899 
0xffffff93b1c8bc50 : 0xffffff800519776b 
0xffffff93b1c8bc90 : 0xffffff80051f336c 
0xffffff93b1c8be00 : 0xffffff8005205fb3 
0xffffff93b1c8bef0 : 0xffffff80052028a6 
0xffffff93b1c8bf60 : 0xffffff800522afd1 
0xffffff93b1c8bfb0 : 0xffffff8004df4b16 

I assume that since the computer underwent a reboot, the addresses translation is useless now since the memory mapping may be different after each booting iteration.

Is there any option to match the relevant method to each address in retrospect, or set an appropriate configuration in advance ? thanks

Answer 1

Yes, you can definitely symbolicate kernel panic traces, but to do it retroactively, you need more information from the panic log than just the raw stack trace. As you say, the addresses only make sense relative to the load addresses.

Apple's official documentation on the topic, TN2063 is a little outdated. The example it gives is from Darwin 9, i.e. OS X 10.5, and things have changed a bit since then, with the introduction of Kernel ASLR and kext UUIDs. I'll try to give you a very quick up to date guide.

1. The Easy Way

If your panic is reproducible, the easiest thing to do is to get the kernel to symbolicate it for you. Using the kernel boot argument keepsyms=1 means the kernel won't discard whatever symbols are stored in the kernel and kext images, and will look up the return pointers in the stack trace in case of a panic.

Just add keepsyms=1 to either the Kernel Flags setting in /Library/Preferences/SystemConfiguration/com.apple.Boot.plist, or to the boot-args NVRAM variable. Reboot, and any subsequent panics will be automatically symbolicated. You can run mangled C++ symbols through the c++filt command line utility to get the proper C++ function signatures. For example,

$ echo __ZN32IOPCIMessagedInterruptController17registerInterruptEP9IOServiceiPvPFvS2_S2_S2_iES2_ | c++filt
IOPCIMessagedInterruptController::registerInterrupt(IOService*, int, void*, void (*)(void*, void*, void*, int), void*)

2. The Manual Way

If you've got an unsymbolicated, mysterious panic you can't seem to reproduce, the easy way isn't much help.

Immediately following the stack trace, look for a section starting with "Kernel Extensions in backtrace:" in the panic log. This will list any kexts involved in the panic, including their load addresses, versions and UUIDs. The addresses are given as a range; the start address is to the left of the -> and following the @. The last address is to the right of the arrow. With this information you should be able to identify the kext in which each code address (the hex number on the right) listed in the stack trace is located.

Except some of them won't match any kext. Except in some odd circumstances, these will be from the kernel itself. The kernel image (kernel or mach_kernel) load address is further down, where it says "Kernel text base:"

Once you know which executable image to look in, the atos command will let you symbolicate each address.

For example, let's say we have this line in a panic:

…
0xffffff8098c1bba0 : 0xffffff7f80c343f2
…

And we also find that:

  Kernel Extensions in backtrace:
     com.apple.iokit.IOPCIFamily(2.9)[BDA92C3B-AD86-33E5-A7F1-1603465350A7]@0xffffff7f80c1a000->0xffffff7f80c4dfff

Notice that 0xffffff7f80c343f2 is greater than (or equal to) 0xffffff7f80c1a000 and less than (or equal to) 0xffffff7f80c4dfff, so the code in question is in IOPCIFamily.

This leads me to the following command (and its output):

$ atos -o /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Extensions/IOPCIFamily.kext/IOPCIFamily -l 0xffffff7f80c1a000 0xffffff7f80c343f2
IOPCIMessagedInterruptController::registerInterrupt(IOService*, int, void*, void (*)(void*, void*, void*, int), void*) (in IOPCIFamily) (IOPCIMessagedInterruptController.cpp:85)

-o specifies the executable file. This is generally in the Contents/MacOS/ subdirectory of the .kext package, but some of Apple's kexts have it directly inside the .kext directory. For functions in the kernel itself, provide the kernel image, e.g. /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.

The -l argument specifies the load address. I.e. start/text base.

Finally, just list all the addresses you want to symbolicate in that file. In this case, just one, but you can list multiple. You can also read them from stdin (if none are listed on the command line).

With this, you should be able to decode your whole trace.

A note on UUIDs

You'll notice that each kext in the trace, and the kernel itself, have a UUID listed. This is handy for making sure you're using the right version for symbolicating. This is the UUID from the LC_UUID loader command in the Mach-O binary file. You can check the UUID for a kext using:

$ otool -l /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Extensions/IOPCIFamily.kext/IOPCIFamily | grep uuid
    uuid BDA92C3B-AD86-33E5-A7F1-1603465350A7

To confirm that the kext being used for symbolication does indeed match the one in the panic. This is great when you're ending up with weird versioning issues, or if you're having trouble with the kext cache.