Java vm slows down with all threads busy with String operations

Question

I am running into a very peculiar issue. My tomcat runs perfectly at about 25% CPU 24/7, but some days my CPU shoots up to 60% and the system grinds to a halt and fails to recover.

When I take a thread dump during the slow down almost all the threads are busy with some kind of String or related operation.

There are no OutOfMemory errors or any exceptions being thrown, all requests are still handled but response times deteriorate to the nth degree where even a sub second request slows down to take 60 seconds and more.

My server config is as follows:

     Ubuntu 12.04.2 LTS     Linux 3.2.0-38-virtual #60-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux 

     java version "1.7.0_13"     Java(TM) SE Runtime Environment (build 1.7.0_13-b20)     Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode) 

     export JAVA_OPTS='-server     -Xms18g -Xmx18g     -XX:MaxPermSize=512m     -XX:ThreadStackSize=512     -XX:NewRatio=1     -XX:SurvivorRatio=4     -XX:+UseConcMarkSweepGC     -XX:+UseParNewGC     -XX:+CMSClassUnloadingEnabled     -Xloggc:/usr/tomcat/logs/gc.log     -XX:+PrintGCDetails     -XX:+PrintGCDateStamps     -XX:+PrintTenuringDistribution     -Dcom.sun.management.jmxremote     -Dcom.sun.management.jmxremote.port=9999     -Dcom.sun.management.jmxremote.authenticate=false     -Dcom.sun.management.jmxremote.ssl=false     -Djava.awt.headless=true' 

Click here to download thread dump. I have removed the bulk of the threads and their stackTraces

Click here to download vmstat log

Click here to download gc log

Any ideas as to the cause of this? Thanks

Answer 1

Try increasing the maximum size of the Code Cache with the following JVM option:

-XX:ReservedCodeCacheSize=256m 

See my answer to another question for the background to this suggestion.