Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why does the JVM of these Tomcat servers perform a full GC hourly?

We run many Tomcat servers and have observed full garbage collections (GCs) are often performed on an hourly basis, particularly when memory usage is relatively low. The precise time appears to be relative to the time the application server was started; if a server is started at 01:13, a full GC is done at 02:13, and the next full GC will occur at 03:13. I haven't been able to find any documentation to explain this behavior.

This is a problem because a server pool started simultaneously all tend to do full GCs at around the same time. If the GC delay is long enough to cause a load-balancer to mark a server as down, the entire application can go offline for a time. It would be better if the full GCs could be distributed across a period so no two servers are doing a full GC at the same time, but I can't find any way to control this behavior.

Has anyone else seen this behavior? Is there any way to influence when these "regular" full GCs happen?

like image 901
user2076909 Avatar asked Feb 15 '13 20:02

user2076909


People also ask

Why is GC taking so long?

If your application's object creation rate is very high, then to keep up with it, the garbage collection rate will also be very high. A high garbage collection rate will increase the GC pause time as well. Thus, optimizing the application to create fewer objects is THE EFFECTIVE strategy to reduce long GC pauses.

What causes major garbage collection in Java?

Excessive garbage collection activity can occur due to a memory leak in the Java application. Insufficient memory allocation to the JVM can also result in increased garbage collection activity. And when excessive garbage collection activity happens, it often manifests as increased CPU usage of the JVM!

What is Java full GC?

Java garbage collection is the process by which Java programs perform automatic memory management. Java programs compile to bytecode that can be run on a Java Virtual Machine, or JVM for short. When Java programs run on the JVM, objects are created on the heap, which is a portion of memory dedicated to the program.

What is GC in Tomcat?

This article provides information on how to enable garbage collection logging with Apache Tomcat, vFabric ERS Tomcat, or vFabric tc Server Runtime on a Linux or UNIX platform. In all these cases, you need to add options to the Java executable, but the method is different for each product.


2 Answers

Your "regular" hourly GCs are probably due to this known bug, "The JreMemoryLeakPreventionListener causes a full GC every hour when gcDaemonProtection=true".

Confirm your Tomcat versions and the value of the gcDaemonProtection property of your JreMemoryLeakPreventionListener (default is true).

The patch was purportedly included in Tomcat v.7.0.28+ and v.6.0.36+.

Either upgrade your server(s), or choose a non-upgrading solution from here, summarized as:

  1. suppress full garbage collections using JVM arg -XX:+DisableExplicitGC
  2. keep the full GCs, but defer to the CMS collector using JVM arg -XX:+ExplicitGCInvokesConcurrent
  3. set <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" gcDaemonProtection="false"/>
  4. Disable the listener

Credit where credit is due; I obtained my initial answer from here.

like image 88
JoshDM Avatar answered Sep 24 '22 03:09

JoshDM


you should be able to change the interval by

-Dsun.rmi.dgc.client.gcInterval=60000 -Dsun.rmi.dgc.server.gcInterval=60000 

take a look here https://docs.oracle.com/cd/E19199-01/817-2180-10/pt_chap5.html

like image 28
grepit Avatar answered Sep 26 '22 03:09

grepit