Howto debug kernel oops on embedded system

Question

I've got a problem with one of the serial ports on an embedded development. /dev/ttyS0, /dev/ttyS2, and /dev/ttyS3 all work fine with no problems. But in some cases accessing /dev/ttyS1 throws the following two 'oops' messages.

I've no idea where to even start tracking down the cause of this, can you help?

1st

Unable to handle kernel NULL pointer dereference at virtual address 0000013c
pgd = c0004000
[0000013c] *pgd=00000000
stopped custom tracer.
Internal error: Oops: 17 [#1] PREEMPT
Modules linked in: macb
CPU: 0    Not tainted  (2.6.24-rc5-rt1 #2)
pc : [<c01a9e60>]    lr : [<c01a9e90>]    psr: 60000093
sp : c3c25f10  ip : 0000012c  fp : c3c25f1c
r10: 00000000  r9 : 00000000  r8 : 00000000
r7 : 000000ac  r6 : 00000000  r5 : 0000012c  r4 : 00000000
r3 : 60000093  r2 : 60000013  r1 : c3c1a340  r0 : 0000012c
Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 0005317f  Table: 23dcc000  DAC: 00000017
Process softirq-high/0 (pid: 4, stack limit = 0xc3c24258)
Stack: (0xc3c25f10 to 0xc3c26000)
5f00:                                     c3c25f2c c3c25f20 c01a9e90 c01a9e20
5f20: c3c25f44 c3c25f30 c0101b84 c01a9e90 c020ab48 c020abf4 c3c25f74 c3c25f48
5f40: c0119960 c0101b78 c3c1a340 c020ac2c 000f4240 00000000 00000004 00000000
5f60: c0205b78 c02059c8 c3c25f98 c3c25f78 c003cf5c c0119560 00000001 fffffffe
5f80: c02059d4 c3c24000 00000001 c3c25fa8 c3c25f9c c003d038 c003cee4 c3c25fd8
5fa0: c3c25fac c003d9ec c003d010 00000032 c3c24000 c02059c8 c003d8a0 00000000
5fc0: 00000000 00000000 00000000 c3c25ff4 c3c25fdc c004cc94 c003d8b0 00000000
5fe0: 00000000 00000000 00000000 c3c25ff8 c003b13c c004cc4c 00000000 00000000
Backtrace:
Function entered at [<c01a9e10>] from [<c01a9e90>]
Function entered at [<c01a9e80>] from [<c0101b84>]
Function entered at [<c0101b68>] from [<c0119960>]
 r5:c020abf4 r4:c020ab48
Function entered at [<c0119550>] from [<c003cf5c>]
Function entered at [<c003ced4>] from [<c003d038>]
 r8:00000001 r7:c3c24000 r6:c02059d4 r5:fffffffe r4:00000001
Function entered at [<c003d000>] from [<c003d9ec>]
Function entered at [<c003d8a0>] from [<c004cc94>]
Function entered at [<c004cc3c>] from [<c003b13c>]
 r6:00000000 r5:00000000 r4:00000000
Code: e592100c e10f2000 e3823080 e121f003 (e59c3010)

2nd

Unable to handle kernel NULL pointer dereference at virtual address 000000bc
pgd = c0004000
[000000bc] *pgd=00000000
Internal error: Oops: 17 [#2] PREEMPT
Modules linked in: macb
CPU: 0    Tainted: G      D  (2.6.24-rc5-rt1 #2)
pc : [<c0101868>]    lr : [<c01161c0>]    psr: 60000013
sp : c3c33f50  ip : c3c33f68  fp : c3c33f64
r10: c0205ab8  r9 : c0205b78  r8 : 00000000
r7 : 00000004  r6 : 00000000  r5 : 000f4240  r4 : c3e3c378
r3 : c3e3c360  r2 : 60000013  r1 : a0000013  r0 : 00000000
Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 0005317f  Table: 23dcc000  DAC: 00000017
Process softirq-tasklet (pid: 9, stack limit = 0xc3c32258)
Stack: (0xc3c33f50 to 0xc3c34000)
3f40:                                     c3e3c378 000f4240 c3c33f74 c3c33f68
3f60: c01161c0 c010186c c3c33f98 c3c33f78 c003cf5c c01161b8 00000020 ffffffdf
3f80: c0205ac4 c3c32000 00000020 c3c33fa8 c3c33f9c c003d078 c003cee4 c3c33fd8
3fa0: c3c33fac c003d9ec c003d050 00000032 c3c32000 c0205ab8 c003d8a0 00000000
3fc0: 00000000 00000000 00000000 c3c33ff4 c3c33fdc c004cc94 c003d8b0 00000000
3fe0: 00000000 00000000 00000000 c3c33ff8 c003b13c c004cc4c ffffffff ffffffff
Backtrace:
Function entered at [<c010185c>] from [<c01161c0>]
 r5:000f4240 r4:c3e3c378
Function entered at [<c01161a8>] from [<c003cf5c>]
Function entered at [<c003ced4>] from [<c003d078>]
 r8:00000020 r7:c3c32000 r6:c0205ac4 r5:ffffffdf r4:00000020
Function entered at [<c003d040>] from [<c003d9ec>]
Function entered at [<c003d8a0>] from [<c004cc94>]
Function entered at [<c004cc3c>] from [<c003b13c>]
 r6:00000000 r5:00000000 r4:00000000
Code: c01f63a4 e1a0c00d e92dd830 e24cb004 (e59030bc)

Answer 1

The best thing you can do is to take a look at Documentation/oops-tracing.txt It explains how you can start debugging this.

The first things you can try is decoding the backtrace. The system.map file should contain the addresses of all functions in your kernel. You should be able to produce a readable backtrace this way.