top shows high user-space CPU utilization but all the process show 0

Question

Any ideas on why the load would be through the roof, top shows each CPU user-space utilization at 85% or higher, yet no process is tagged with any CPU utilization?

First time I've ever seen this. We've got a server that has a load average of 20, and the break down on each CPU is extremely high (>85%), yet none of the processes in the list have any CPU % utilization. Here's a sample, and yes, this is with the processes sorted from highest to lowest CPU utilization - every once in a blue moon we'll actually see a process that shows "1" as the CPU %.

This is a mysql server box.

top - 16:08:48 up 185 days, 15:27,  5 users,  load average: 20.60, 17.20, 13.17
Tasks: 221 total,   2 running, 219 sleeping,   0 stopped,   0 zombie
Cpu0  : 90.3%us,  5.8%sy,  0.0%ni,  3.2%id,  0.6%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 89.1%us,  5.8%sy,  0.0%ni,  4.5%id,  0.6%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 86.4%us,  5.0%sy,  0.9%ni,  6.9%id,  0.6%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  : 88.1%us,  5.8%sy,  1.3%ni,  4.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  : 89.8%us,  7.2%sy,  0.0%ni,  1.3%id,  0.0%wa,  0.0%hi,  1.6%si,  0.0%st
Cpu5  : 91.0%us,  2.6%sy,  0.0%ni,  5.4%id,  1.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  : 91.8%us,  4.6%sy,  0.3%ni,  3.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 78.8%us, 14.4%sy,  1.3%ni,  5.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  12326280k total, 12147144k used,   179136k free,   158512k buffers
Swap:  8420344k total,        0k used,  8420344k free, 10731260k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                             
22691 root      20   0 19348 1536 1068 R    1  0.0   0:00.29 top                                                                                                                                                                 
 8723 root      20   0 19356 1556 1064 S    0  0.0   0:01.51 top                                                                                                                                                                 
21562 zabbix    25   5 64000 1224  656 S    0  0.0  54:17.09 /usr/sbin/zabbix_agentd                                                                                                                                             
21566 zabbix    25   5 63976 1132  524 S    0  0.0 151:05.16 /usr/sbin/zabbix_agentd                                                                                                                                             
25165 root      20   0 18700 2744 1840 R    0  0.0   0:00.01 perl /usr/local/bin/mk-heartbeat -D utility --interval 1 --update -h rs4 --password                                                                       
    1 root      20   0 23708 1184  596 S    0  0.0   0:33.56 /sbin/init                                                                                                                                                          
    2 root      20   0     0    0    0 S    0  0.0   0:00.01 [kthreadd]                                                                                                                                                          
    3 root      RT   0     0    0    0 S    0  0.0   0:34.65 [migration/0]           

Answer 1

Load average doesn't show CPU use -- it shows you how many runnable processes are on the "run queue": waiting to run or blocked waiting for disk IO. You won't see them taking up CPU time -- instead, if you look for processes in state R or D, that will show you roughly the tasks that are involved in the load average calculation.

Answer 2

I see this occasionally when one of the hardware devices (DVD or Wifi modem) are having problems. The driver is resetting or reinitializing the hardware with busy waits. No process is charged for the time, but the system acts really busy, no matter how low the load average is. There is completely coincident evidence when conditions return to normal in the system log.