G1GC very high GC count and CPU, very frequency GCs that kill performance

Question

I've recently switched my Java application from CMS + ParNew to G1GC. What I observed when I did the switch is the CPU usage went higher and the GC count + pause time went up as well. My JVM flags before the switched were

 java -Xmx22467m -Xms22467m -XX:NewSize=11233m -XX:+UseConcMarkSweepGC -XX:AutoBoxCacheMax=1048576 -jar my-application.jar

After the switch my flags are:

java -Xmx22467m -Xms22467m -XX:+G1GC -XX:AutoBoxCacheMax=1048576 -XX:MaxGCPauseMillis=30 -jar my-application.jar

I followed Oracle's Best Practices http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html

Do not Set Young Generation Size

And did not set the young generation size. However I am suspecting that the young generation size is the problem here. What I see is the heap usage is fluctuating between ~6 - 8 GB.

Whereas before, with CMS and Par New there the memory usage grew between 4-16 GB and only then I saw a GC:

I am not sure I understand why with G1GC the GC is so frequent. I am not sure what I'm missing when it comes to GC tuning with G1GC.

I'm using Java 8 : ava version "1.8.0_144" Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

I appreciate your help.

UPDATE: A bit more information about those pauses: As you can see all those pauses are G1New, and seemingly they are as long as my target pause time, which is 30ms. When I look at the ParNew pauses before the switch to G1GC, this is how it looked like: So they are also all young gen collections (ParNew) but they are less frequent and shorter, because they happen only when the heap usage gets to around 14GB (according to the graph)

I am still clueless why the G1New happen so early (in terms of heap usage)

Update 2 I also noticed that NewRatio=2, I don't know if G1GC is respecting that, but that would mean that my New Gen is capped at 7GB. Could that be the reason?

Update 3 Adding G1GC GC logs: https://drive.google.com/file/d/1iWqZCbB-nU6k_0-AQdvb6vaBSYbkQcqn/view?usp=sharing

Answer 1

I was able to see that the time spent in copying objects is very significant. Looks like G1GC has 15 generations by default before the object is promoted to Tenured Generation. I reduced it to 1 (-XX:MaxTenuringThreshold=1)

Also I don't know how to confirm it in the logs, however visualizing the GC log I saw that the young generation is constantly being resized, from minimum size to maximum size. I narrowed down the range and that also improved the performance.

Looking here https://docs.oracle.com/javase/9/gctuning/garbage-first-garbage-collector-tuning.htm#JSGCT-GUID-70E3F150-B68E-4787-BBF1-F91315AC9AB9 I was trying to figure out if coarsenings is indeed an issue. But it simply says to set gc+remset=trace which I do not understand how to pass to java in command line, and if it's even available in JDK 8. I increased the XX:G1RSetRegionEntries a bit just in case.

I hope it helps to the future G1GC tuner and if anyone else has more suggestions that would be great.

What I still see is that [Processed Buffers] is still taking a very long time in young evacuations, and [Scan RS] is very long in mixed collections. Not sure why