Is smaller eden space causing tenure space to increase constantly?

Question

I will first give some details of the objects that we create. And will ask a question (in the end).

I am using 64-but JVM on linux.

My jvm options are: -Xmx6g -Xms3g -XX:MaxPermSize=256m

The profiler (jprofiler) indicates 2 places consuming lot of memory: 1) Here, about 4 strings (each 110KB) per second are created and StringUtils.replace is performed on every string. Some other string assignments, etc are also performed, but the profiler clearly indicates StringUtils.replace is what is holding most of the memory. 2) Here, an object (retained size if 3.5MB) containing 4 objects (of a single class) is created one per minute and (re)placed into a static concurrenthashmap (so map contains only 1 object at any time). All the parent object has is these 4 child objects and nothing else. Each of those 4 objects contain 10K arraylists, 10K Date and other strings, etc. I am explicitly assigning the old parent object (the one I get from map.replace()) to null and explicitly clearing (ArrayList.clear()) out the ArrayList, etc inside those 4 child objects.

The memory behavior is: The memory usage keeps increasing and at one point major collection (2GB) happens. minor collection (700MB) also happen about once a minute.

Question: Do you think #1 is actually causing the eden space to be full and so JVM is pushing objects in #2 to tenure space and so the memory usage keeps increasing until major colection happens? For diagnosis purpose, I changed the parent object in #2 to have retained size of 500KB (7 times lesser, also 1.4K instances of Date, ArrayList, etc), and I still see the same memory behavior.

I changed the options to -Xmx12g -Xms3g -XX:MaxPermSize=256m -XX:NewSize=8g Now, I see lesser # of objects in #2 in memory I see slower increase of tenure space / old gen. Its good that those objects in #2 are being collected quickly now, but CPU usage is extremely high because of minor collections now are huge (8G) and happen every 2 mins.

My goal is quick collection of all the objects (so I don't see memory usage increasing constantly), but not make the minor collections too big.

Any suggestions on what approach I should take?

Answer 1

The cost of collecting the eden space is proportional to the number of objects in the eden space which are retained. As the cost is high, it suggest a large proportion of the objects are being retained longer than 2 minutes.

As you are creating around 6 GB of objects every two minutes (The NewSize includes two survivor spaces), or around 3 GB per minute where is the bulk of this coming from as the sizes you suggest are only a few MB per minute.

I think a better undersatnding of where the 3 GB is coming from and reducing it would be useful as it appears you only need to keep about 10 MB per minute.

BTW: If you were to create 10 MB of garbage per minute, you would only create 6 GB in ten hours and you wouldn't need any minor or mjor collections in that time.

If I run

public class GCLikeMad {
    public static void main(String... args) {
        for(Long l = 0L; l < Integer.MAX_VALUE * 10L; l++);
    }
}

with -XX:NewSize=256m I see

[GC 295664K->240K(2121792K), 0.0011190 secs]
[GC 286832K->240K(2113216K), 0.0011510 secs]
[GC 278384K->240K(2105216K), 0.0012900 secs]
[GC 270256K->240K(2097280K), 0.0010110 secs]
[GC 262448K->240K(2257280K), 0.0013450 secs]

---

with -XX:newSize=8g -verbosegc I see

[GC 6291664K->320K(7345408K), 0.0017790 secs]
[GC 6291776K->272K(7345408K), 0.0021570 secs]
[GC 6291728K->272K(7345408K), 0.0014490 secs]
[GC 6291728K->240K(8393600K), 0.0016680 secs]
[GC 8388080K->208K(8393600K), 0.0018730 secs]

---

with -XX:NewSize=30g -mx31g -XX:SurvivorRatio=100 -verbosegc At this point 32-bit references and only one NUMA region of memory is used.

[GC 30840864K->256K(31154304K), 0.0014270 secs]
[GC 30840832K->256K(31154304K), 0.0014730 secs]
[GC 30840832K->224K(31154304K), 0.0016500 secs]
[GC 30840800K->272K(31154304K), 0.0015740 secs]
[GC 30840848K->272K(31462336K), 0.0015280 secs]

---

with -XX:NewSize=128g -mx130g -XX:SurvivorRatio=100 -verbosegc it starts to make a difference. Note: At this point multiple NUMA regions of memory and 64-bit references are being used. This is collecting every 2 minutes.

[GC 131586048K->320K(132907264K), 0.0042360 secs]
[GC 131586368K->1376K(132907264K), 0.0058030 secs]
[GC 131587424K->1440K(132907264K), 0.0034110 secs]

At this point it had create 20 billion Long objects with only three collections.

I would say a 128 GB Eden space is huge. :D

---

My point being that increasing the eden size, doesn't increase the GC time alone. It the increased number of objects being retained which increases the time taken.