Debugging linux kernel panic

Question

I am new to kernel debugging.Below is the last kernel message. According to my understanding it is trying to access the null pointer. But i do not understand why it is shutting down the cup 0.

Unable to handle kernel paging request at virtual address 40025694
[37655.473976] pgd = d7138000
[37655.474007] [40025694] *pgd=9ec1a831, *pte=bf2e659d, *ppte=00000000
[37655.474038] Internal error: Oops: 17 [#1] PREEMPT SMP
[37655.474038] Modules linked in:
[37655.474068] CPU: 1    Not tainted  (3.0.21-g572d9be-00004-g4ec4db2 #1)
[37655.474099] PC is at vector_swi+0x28/0x88
[37655.474129] LR is at 0x40025698
[37655.474160] pc : c01065a8    lr : 40025698    psr : 60000093
[37655.474160] sp : d6eaffb0  ip : 4062c18c  fp : 5ed97c24
[37655.474190] r10: 58702b64  r9 : 5ea56f98  r8 : 20000010
[37655.474221] r7 : 000000a8  r6 : 41b13530  r5 : 4062c270  r4 : 4062c140
[37655.474251] r3 : 00000000  r2 : ffffffff  r1 : 00000001  r0 : 5ed97bd0
[37655.474251] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
[37655.474282] Control: 10c5787d  Table: 9ec3806a  DAC: 00000015
[37655.474282] 
[37655.474282] PC: 0xc0106528:
[37655.474312] 6528  e31100ff 1afffff0 e59d1040 e5bde03c e16ff001 f57ff01f e95d7fff e1a00000
[37655.474343] 6548  e28dd00c e1b0f00e eb01ae0e e1a096ad e1a09689 e5991000 e3a08001 e3110c01
[37655.474373] 6568  0affffec e1a0100d e3a00001 eb0006ef eaffffe8 e320f000 e24dd048 e88d1fff
[37655.474404] 6588  e28d803c e9486000 e14f8000 e58de03c e58d8040 e58d0044 e3180020 13a0a000
[37655.474434] 65a8  051ea004 e59fc0ac e59cc000 ee01cf10 f1080080 e1a096ad e1a09689 e28f809c
[37655.474465] 65c8  e3daa4ff 122a7609 159f808c e599a000 e92d0030 e31a0c01 1a000008 e3570f5e
[37655.474465] 65e8  e24fee13 3798f107 e28d1008 e3a08000 e357080f e2270000 2a001084 ea022377
[37655.474526] 6608  e1a02007 e28d1008 e3a00000 eb0006c7 e28fe014 e1a07000 e28d1008 e3570f5e
[37655.474526] 
[37655.474526] SP: 0xd6eaff30:
[37655.474556] ff30  c004815c c004a2c0 c0c02080 0000000a c0d8a580 c017a2f0 c1ef1984 00000012
[37655.474587] ff50  ffffffff d6eaff9c 41b13530 000000a8 20000010 c010602c 5ed97bd0 00000001
[37655.474617] ff70  ffffffff 00000000 4062c140 4062c270 41b13530 000000a8 20000010 5ea56f98
[37655.474648] ff90  58702b64 5ed97c24 4062c18c d6eaffb0 40025698 c01065a8 60000093 ffffffff
[37655.474678] ffb0  5ed97bd0 00000001 ffffffff 00000000 4062c140 4062c270 41b13530 000000a8
[37655.474678] ffd0  5ed97bd0 5ea56f98 58702b64 5ed97c24 4062c18c 5ed97bc8 406283d7 40025698
[37655.474709] fff0  20000010 5ed97bd0 a8afc821 a8afcc21 e033cfc0 40eb3000 40eb6000 d6eb0058
[37655.474739] 0010  d75a9d68 0000079f 00100073 d6eb017d d6eb00cc d6e98f94 00000000 00000000
[37655.474770] Process UEventObserver (pid: 675, stack limit = 0xd6eae2f0)
[37655.474770] Stack: (0xd6eaffb0 to 0xd6eb0000)
[37655.474801] ffa0:                                     5ed97bd0 00000001 ffffffff 00000000
[37655.474801] ffc0: 4062c140 4062c270 41b13530 000000a8 5ed97bd0 5ea56f98 58702b64 5ed97c24
[37655.474831] ffe0: 4062c18c 5ed97bc8 406283d7 40025698 20000010 5ed97bd0 a8afc821 a8afcc21
[37655.474831] Code: e58d8040 e58d0044 e3180020 13a0a000 (051ea004) 
[37655.474862] ---[ end trace 2416079997dfe426 ]---
[37655.474862] Kernel panic - not syncing: Fatal exception
[37655.474892] [<c010cdec>] (unwind_backtrace+0x0/0x12c) from [<c078872c>] (panic+0x80/0x1a4)
[37655.474923] [<c078872c>] (panic+0x80/0x1a4) from [<c010a578>] (die+0x1d4/0x21c)
[37655.474953] [<c010a578>] (die+0x1d4/0x21c) from [<c0111510>] (__do_kernel_fault+0x64/0x84)
[37655.474953] [<c0111510>] (__do_kernel_fault+0x64/0x84) from [<c0111798>] (do_page_fault+0x268/0x288)
[37655.474984] [<c0111798>] (do_page_fault+0x268/0x288) from [<c0100340>] (do_DataAbort+0x134/0x1a4)
[37655.474984] [<c0100340>] (do_DataAbort+0x134/0x1a4) from [<c010602c>] (__dabt_svc+0x4c/0x60)
[37655.475014] Exception stack(0xd6eaff68 to 0xd6eaffb0)
[37655.475014] ff60:                   5ed97bd0 00000001 ffffffff 00000000 4062c140 4062c270
[37655.475014] ff80: 41b13530 000000a8 20000010 5ea56f98 58702b64 5ed97c24 4062c18c d6eaffb0
[37655.475045] ffa0: 40025698 c01065a8 60000093 ffffffff
[37655.475045] [<c010602c>] (__dabt_svc+0x4c/0x60) from [<c01065a8>] (vector_swi+0x28/0x88)
[37655.475075] CPU0: stopping
[37655.475075] [<c010cdec>] (unwind_backtrace+0x0/0x12c) from [<c010b438>] (handle_IPI+0x100/0x1d4)
[37655.475106] [<c010b438>] (handle_IPI+0x100/0x1d4) from [<c010044c>] (gic_handle_irq+0x9c/0xac)
[37655.475106] [<c010044c>] (gic_handle_irq+0x9c/0xac) from [<c0106094>] (__irq_svc+0x54/0x80)
[37655.475136] Exception stack(0xc874bc68 to 0xc874bcb0)
[37655.475136] bc60:                   d3cf4948 cd6dbd54 0000001f 00000000 b1d2a59d 5ea1f000
[37655.475136] bc80: d3cf4948 5ea1f000 00000000 c874a000 d2b9207c d7e55670 5ea1f02b c874bcb0
[37655.475167] bca0: c01eb894 c010bbb4 60000013 ffffffff
[37655.475167] [<c0106094>] (__irq_svc+0x54/0x80) from [<c010bbb4>] (flush_tlb_page+0x8c/0x98)
[37655.475197] [<c010bbb4>] (flush_tlb_page+0x8c/0x98) from [<c01eb894>] (ptep_clear_flush+0x30/0x38)
[37655.475197] [<c01eb894>] (ptep_clear_flush+0x30/0x38) from [<c01e743c>] (try_to_unmap_one+0xc4/0x3a4)
[37655.475228] [<c01e743c>] (try_to_unmap_one+0xc4/0x3a4) from [<c01e77a0>] (try_to_unmap_file+0x84/0x4b4)
[37655.475228] [<c01e77a0>] (try_to_unmap_file+0x84/0x4b4) from [<c01e80b0>] (try_to_unmap+0x34/0x4c)
[37655.475258] [<c01e80b0>] (try_to_unmap+0x34/0x4c) from [<c01d28dc>] (shrink_page_list+0x258/0x78c)
[37655.475258] [<c01d28dc>] (shrink_page_list+0x258/0x78c) from [<c01d31b4>] (shrink_inactive_list+0x1e0/0x330)
[37655.475289] [<c01d31b4>] (shrink_inactive_list+0x1e0/0x330) from [<c01d3704>] (shrink_zone+0x400/0x588)
[37655.475289] [<c01d3704>] (shrink_zone+0x400/0x588) from [<c01d3fd8>] (kswapd+0x594/0x970)
[37655.475319] [<c01d3fd8>] (kswapd+0x594/0x970) from [<c018e04c>] (kthread+0x80/0x88)
[37655.475319] [<c018e04c>] (kthread+0x80/0x88) from [<c01075c0>] (kernel_thread_exit+0x0/0x8)

Any pointer so that i can go forward.

Thanks

Answer 1

If your not able to reproduce the oops the only real recourse you have is to look at the objdump of the kernel and match up the CPU state dumped in the oops with the location of the variables. There used to be a guide (written by Linus himself!) in the kernel tree: Documentation/oops-tracing.txt. However it seems the Kernel User Manual now has more upto date documentation rendered from the source tree.