Java server cpu usage at 100% after two days continous running with about 110 users

Question

I have a tomcat 6.0.20, apr 1.2, jdk 1.6.0_15 with mysql 5.1.38 running on a rhel box with 4 GB ram. There is one simple jsp/servlet application on it with 5 users, one struts 1.2.0.9 with 64 users on it and one struts 2.0 application with 35 users on it. The struts 2.0 users make an entry every second, about 900 entries in a day. I am also using toplink for persistence in the last two applications. I have declared all non referenced objects to null, in the code, have applied production values for config files from the struts 2 site and from the tomcat site. Applied thread caching in mysql, reduce wait_timeout and interactive_timeout to be equivalent to session timeout of tomcat.Increased file descriptors in linux. Reworked queries. Examined the thread dump, watched gc stats, applied above changes on basis of this,

YET still facing "java.lang.OutOfMemoryError" error.

at different times its for different things, sometimes its Servlet.service(), sometimes its image.servlet, sometimes it jasper that causes it.

extremely frustrtating, as the errors are not constant but keep changing over time

Any help please would be greatly appreciated!!!

JAVA_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+CMSParallelRemarkEnabled (tomcat manager reports 34 mb empty so have not used permsize, mx and mn etc.)

persistence.xml

 <property name="toplink.jdbc.url" value="jdbc:mysql://localhost:3306/dbname?autoReconnect=false"/>

server.xml

<Connector port="80" protocol="HTTP/1.1" connectionTimeout="2000" redirectPort="8443" compression="on" compressableMimeType="application/octet-stream,text/html,text/xml,text/plain,application/x-javascript,image/gif,text/css,image/gif,application/vnd.ms-excel,application/pdf" enableLookups="false"/>

context.xml

<Context reloadable="false" delegate="false" privileged="false">

Answer 1

First, you must make sure which process is eating all the CPU. Is it really the java program? Try top(1) to figure this out if you haven't already.

If you have and you're sure that it's the Java program, enable remote debugging. When the CPU boils over next time, connect and make sure that no thread is in an infinite loop.

If that's not the case, you're out of memory (no matter what the tomcat manager says). Start a JMX console and check the various memory spaces. My guess is that the permgen is pretty full (are there many big JARs on your classpath? Or are you using cglib somewhere?) When that happens (and since you have perm gen GC enabled), the Java VM will try to free memory in the perm gen space all the time.

Unless you are using something which creates class files at runtime (a scripting language like Groovy, for example), that will not work: In normal Java programs, classes can never be GC'd. If you still do it, the GC will run and run and achieve nothing but burn all CPU power.