Menu
I am running into a very peculiar issue. My tomcat runs perfectly at about 25% CPU 24/7, but some days my CPU shoots up to 60% and the system grinds to a halt and fails to recover.
When I take a thread dump during the slow down almost all the threads are busy with some kind of String or related operation.
There are no OutOfMemory errors or any exceptions being thrown, all requests are still handled but response times deteriorate to the nth degree where even a sub second request slows down to take 60 seconds and more.
My server config is as follows:
Ubuntu 12.04.2 LTS Linux 3.2.0-38-virtual #60-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux
java version "1.7.0_13" Java(TM) SE Runtime Environment (build 1.7.0_13-b20) Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
export JAVA_OPTS='-server -Xms18g -Xmx18g -XX:MaxPermSize=512m -XX:ThreadStackSize=512 -XX:NewRatio=1 -XX:SurvivorRatio=4 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/tomcat/logs/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.awt.headless=true'
Click here to download thread dump. I have removed the bulk of the threads and their stackTraces
Click here to download vmstat log
Click here to download gc log
Any ideas as to the cause of this? Thanks
911
Each software thread requires virtual memory for its stack and private data structures. As with caches, time slicing causes threads to fight each other for real memory and thus hurts performance. In extreme cases, there can be so many threads that the program runs out of even virtual memory.
4.2. On Windows machines, there's no limit specified for threads. Thus, we can create as many threads as we want, until our system runs out of available system memory.
Whatever condition it is, jvm is still the vm that has slowest startup. You may find it fast enough on certain grade of machine and for certain size of application. But still its startup time is comparably inarguably slow. I think GUI apps are primarily what gave Java the impression of being slow...
i)It may easily lead to Deadlock state. ii)Having too many threads will have an impact in the time complexity of the problem. iii)Another problem arises when a time slice expires for a thread holding a lock.
Try increasing the maximum size of the Code Cache with the following JVM option:
-XX:ReservedCodeCacheSize=256m See my answer to another question for the background to this suggestion.
157
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
