Analyzing Mac OS X kernel core dumps part 2

Question

I am moved a little bit since last post Infinitely listen to jabber events?

Now I got a correct core dump (it seems that all previous were incorrect) and I can have the backtrace, but I cant load the symbols for this backtrace

So, what I do is

lldb
target create -d -c ~/Downloads/core-xnu-2050.48.12-10.41.20.195-71371911 /Volumes/KernelDebugKit/mach_kernel
bt

(lldb) bt
* thread #1: tid = 0x0000, 0xffffff8009abcecb, stop reason = signal SIGSTOP
* frame #0: 0xffffff8009abcecb
frame #1: 0xffffff8009a1d636
frame #2: 0xffffff8009d4e45f
frame #3: 0xffffff8009d4cf7d
frame #4: 0xffffff8009d9e585
frame #5: 0xffffff8009d59942
frame #6: 0xffffff8009d58ebd
frame #7: 0xffffff7f8b28c299
frame #8: 0xffffff8009d48dc1
frame #9: 0xffffff8009afc6ee
frame #10: 0xffffff8009b103ee
frame #11: 0xffffff8009b02bc9
frame #12: 0xffffff8009b03394
frame #13: 0xffffff8009de97ba

but no symbols (although I loaded one from KernelDebugKit/mach_kernel). Moreover, when I try to run paniclog, for example, I got the following output

(lldb) paniclog

************ LLDB found an exception ************
There has been an uncaught exception. A possible cause could be that remote connection has been disconnected.
However, it is recommended that you report the exception to lldb/kernel debugging team about it.
************ Please run 'xnudebug debug enable' to start collecting logs. ************

Traceback (most recent call last):
  File "/Volumes/KernelDebugKit/mach_kernel.dSYM/Contents/Resources/Python/lldbmacros/xnu.py", line 107, in _internal_command_function
    obj(cmd_args=stream.target_cmd_args)
  File "/Volumes/KernelDebugKit/mach_kernel.dSYM/Contents/Resources/Python/lldbmacros/xnu.py", line 527, in ShowPanicLog
    panic_buf_end = unsigned(kern.globals.debug_buf_ptr)
  File "/Volumes/KernelDebugKit/mach_kernel.dSYM/Contents/Resources/Python/lldbmacros/core/cvalue.py", line 345, in unsigned
    return val._GetValueAsUnsigned()
  File "/Volumes/KernelDebugKit/mach_kernel.dSYM/Contents/Resources/Python/lldbmacros/core/cvalue.py", line 295, in _GetValueAsUnsigned
    raise ValueError("Failed to read unsigned data. "+ str(self._sbval19k84obscure747) +"(type =" + str(self._sbval19k84obscure747_type) + ") Error description: " + serr.GetCString())
ValueError: Failed to read unsigned data. (char *) debug_buf_ptr = <mach_kernel[0xffffff8000801170] can't be resolved, mach_kernel in not currently loaded>
(type =char *) Error description: could not resolve value
(lldb) 

So, what this the correct way to analyze this type of dumps? Articles or anything? =)

Answer 1

What version of lldb are you using? This looks like it should work - Xcode 5.x's lldb is definitely expected to work, earlier releases may not (I can't remember exactly when the kernel core file debugging support was fnished - but I think Xcode 5's lldb was the start of it).

When lldb starts examining a core file, it searches for a kernel binary in the core file. If it finds one, it switches from "user debug" to "kernel debug" mode (specifically - it selects the DynamicLoader and Platform plugins for kernel debugging).

Once you've checked that you're using a recent lldb (e.g. lldb-310.2.x for the latest Xcode 5.x update), you can try running lldb on the core file directly without specifying the kernel binary as a test -

% lldb -c core-xnu-blahblahwhatever--53821b67 
Kernel UUID: 9FEA8EDC-B629-3ED2-A1A3-6521A1885953
Load Address: 0xffffff802c400000

When you see the Kernel UUID: and Load Address: lines, that tells you that lldb found a kernel image in the core file. You can also use the platform status command to confirm which platform is selected:

(lldb) pla sta
  Platform: darwin-kernel
 Connected: no
 Debug session type: Mac OS X kernel debugging
 Kext directories: [ 0] "/System/Library/Extensions"
 Kext directories: [ 1] "/Library/Extensions"
 Kext directories: [ 2] "/Applications/Xcode.app/Contents/Symbols"
 Total number of kexts indexed: 818
(lldb) 

Of course you can't do real kernel debugging without the kernel binary - just a quick tip, you can specify the core file and the binary on the command line,

% lldb -c core-xnu-blahblahwhatever--53821b67 /Volumes/KernelDebugKit/mach_kernel

The addresses in your backtrace look like a kernel core file session, but for some reason the lldb you're using isn't finding the kernel in there.