Elasticsearch threads behaves different from each other

Question

Elasticsearch runs on my low-configured system which has 4G memory and 4 cores CPUs. I get high-cpu usage problem with ES. Even after closing analyzer(s), reduce threads size and etc.

While anaylzing the situation I got a stack trace for Elasticsearch and see there are hundreds of threads, it is defined by config NP, but only some of them running and only one thread has much percentage of CPU time.

Here is stack trace:

top - 09:51:44 up 1 day,  1:46,  2 users,  load average: 4.94, 5.35, 5.29
Tasks: 684 total,   2 running, 682 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.2%us,  1.1%sy,  0.8%ni, 83.7%id,  6.7%wa,  0.1%hi,  0.4%si,  0.0%st
Mem:   4043340k total,  3466748k used,   576592k free,    30272k buffers
Swap:  4192960k total,   410704k used,  3782256k free,   465868k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                           
25134 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25136 root      20   0 1086m 978m  14m S  0.0 24.8   0:03.34 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25137 root      20   0 1086m 978m  14m S  0.0 24.8   6:43.82 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25138 root      20   0 1086m 978m  14m S  0.0 24.8   6:43.17 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25139 root      20   0 1086m 978m  14m S  0.0 24.8   6:43.05 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25140 root      20   0 1086m 978m  14m S  0.0 24.8   6:40.66 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25141 root      20   0 1086m 978m  14m S 15.3 24.8 204:53.20 /usr/bin/java -Xms808m -Xmx808m -Xss256k
25143 root      20   0 1086m 978m  14m S  0.0 24.8  10:47.46 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25144 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.37 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25145 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.04 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25146 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.20 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25147 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25148 root      20   0 1086m 978m  14m S  0.0 24.8   0:19.46 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25149 root      20   0 1086m 978m  14m S  0.0 24.8   0:24.54 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25150 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25151 root      20   0 1086m 978m  14m S  0.0 24.8   0:16.63 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25156 root      20   0 1086m 978m  14m S  0.0 24.8   0:03.07 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25159 root      20   0 1086m 978m  14m S  0.3 24.8   3:46.78 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25201 root      20   0 1086m 978m  14m S  0.0 24.8   0:01.07 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25202 root      20   0 1086m 978m  14m S  0.0 24.8   0:01.16 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25205 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25206 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.54 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25207 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.54 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25208 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.52 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25209 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.56 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25210 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.54 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25211 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.55 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25212 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.53 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25213 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.55 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25214 root      20   0 1086m 978m  14m S  0.0 24.8   0:02.74 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25215 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.55 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25216 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.54 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25217 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.55 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25218 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.54 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25219 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.55 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25220 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.54 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25221 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.52 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25222 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.55 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25223 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25233 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25241 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25245 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25249 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25252 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25259 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.00 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25264 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.24 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25266 root      20   0 1086m 978m  14m S  0.7 24.8   0:40.80 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25269 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.44 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25276 root      20   0 1086m 978m  14m S  0.0 24.8   0:37.28 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25278 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.80 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25280 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.04 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25318 root      20   0 1086m 978m  14m S  0.0 24.8   0:15.87 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25329 root      20   0 1086m 978m  14m S  0.0 24.8   0:09.73 /usr/bin/java -Xms808m -Xmx808m -Xss256k 
25333 root      20   0 1086m 978m  14m S  0.0 24.8   0:00.01 /usr/bin/java -Xms808m -Xmx808m -Xss256k

As it is seem on code block, the thread with id 25141 has much more than others and some of them has not been used at least once.

Why does it occurrs and what should i do to prevent high CPU usage.

Answer 1

Some Threads do more work than others. The one(s) doing network communication always generate load, threads doing querys may generate high load and so on. As far as i know, lucene merges are quite CPU hungry.