Tomcat java.lang.OutOfMemoryError: GC overhead limit exceeded

Question

We trying to migrate our application to Tomcat 7.0 from OC4J. The application works fine with OC4J but, in tomcat the performance gets affected when running load testing with 10 users. We get these errors and the application doesnt respond anymore.

---java.lang.OutOfMemoryError: GC overhead limit exceeded 

Exception in thread "ajp-bio-8009-exec-231" java.lang.OutOfMemoryError: GC overhead limit exceeded 
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: GC overhead limit exceeded 

Exception in thread "ajp-bio-8009-exec-236" java.lang.OutOfMemoryError: GC overhead limit exceeded 
Exception in thread "ajp-bio-8009-exec-208" java.lang.OutOfMemoryError: GC overhead limit exceeded 
java.lang.OutOfMemoryError: GC overhead limit exceeded 

java.lang.OutOfMemoryError: GC overhead limit exceeded 

Exception in thread "Thread-33" java.lang.OutOfMemoryError: GC overhead limit exceeded 

Exception in thread "ajp-bio-8009-exec-258" java.lang.OutOfMemoryError: GC overhead limit exceeded 

java.lang.OutOfMemoryError: GC overhead limit exceeded

We tried JAVA_OPTS="-Xms4096m -Xmx8192m. Looks like we still get the error. Please suggest the possible options that we could try..

Garbage collection logs

GC invocations=593 (full 539): PSYoungGen total 701120K, used 374720K [0x00000007aaab0000, 0x00000007eaf60000, 0x0000000800000000) eden space 374720K, 100% used [0x00000007aaab0000,0x00000007c18a0000,0x00000007c18a0000) from space 326400K, 0% used [0x00000007d70a0000,0x00000007d70a0000,0x00000007eaf60000) to space 339328K, 0% used [0x00000007c18a0000,0x00000007c18a0000,0x00000007d6400000) ParOldGen total 2796224K, used 2796223K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000) object space 2796224K, 99% used [0x0000000700000000,0x00000007aaaaffe8,0x00000007aaab0000) PSPermGen total 50688K, used 50628K [0x00000006fae00000, 0x00000006fdf80000, 0x0000000700000000) object space 50688K, 99% used [0x00000006fae00000,0x00000006fdf713a8,0x00000006fdf80000) 4482.450: [Full GC [PSYoungGen: 374720K->339611K(701120K)] [ParOldGen: 2796223K->2796222K(2796224K)] 3170943K->3135834K(3497344K) [PSPermGen: 50628K->50628K(50688K)], 1.4852620 secs]

Answer 1

GC Overrhead Limit exceeded usually implies the application is 'leaking' memory some where. Your heap is high in memory use, like 98/99% and a full GC will retrieve maybe a percent or two. What that will do is it will spend most of the time GCing. The JVM checks to see how often it has spent GCing and if it exceeds some limit this error is what is thrown.

To resolve it you will need to check where the leak is occurring. Do this by getting a heap dump. You can use jmap for that. Once you get it, you should see the % of the heap will probably belong mostly to one set of objects

We tried JAVA_OPTS="-Xms4096m -Xmx8192m. Looks like we still get the error. Please suggest the possible options that we could try..

That's a lot and only delays the inevitable.

---

Edit from you update

As I expected your OldGen space is at 99% with little reclaimed. OldGen space is where all long lived object will be placed. Since you are not reclaiming some memory all of those objects will eventually be placed into OldGen and you will run out of memory.

What's worth reading are the two lines here:

ParOldGen total 2796224K, used 2796223K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000) object space 2796224K, 99% 

Full GC [PSYoungGen: 374720K->339611K(701120K)] [ParOldGen: 2796223K->2796222K(2796224K)] 3170943K->3135834K(3497344K) 

Like I mentioned, OldGen is at 99% and a Full GC only reclaims 1KB YounGen and 35KB OldGen. It will have to GC almost immediately again. It should be GCing GBs at this point.

So

Get a heap dump and find out what the greatest offender here is. Investigate where these objects are being created and why they are not becoming unreachable.

If you have any other questions about how/where or why let me know, but there is nothing else I can tell you at this point.

Answer 2

The detail message "GC overhead limit exceeded" indicates that the garbage collector is running all the time and Java program is making very slow progress.

Can be fixed in 2 ways 1) By Suppressing GC Overhead limit warning in JVM parameter Ex- -Xms1024M -Xmx2048M -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit

-UseGCOverheadLimit - Parameter is used to suppress GCOverHead.

2) By identifying memory leak