Objects are constantly promoted to the Old Gen

Question

I'm working with a application which provides REST API. It works with GET requests only and usually spends up to 100ms to handle even the most heavy requests.

Recently we started facing a problem that from time to time heap gets fullfilled and full GC takes a lot of time and it really affects our clients.

• Heap size - 5GB
• Concurrent requests per second - up to 50
• Each request consumes up to 1-2MB(There are few requests that consume 12 MB of heap)
• Application uses Paralel GC for both Old and young geerations.
• JDK 1.7

From my point of view this application should promote some object in the Old Gen at the very beginning of work and after it all new objects should get removed from Eden or Survivor spaces. Thus full GC shoulld never come. However some objects keep get promoted into Old Gen and it's really confusing for me.

Several additional findings:

• We enabled JVM flag -XX:+PrintTenuringDistribution and according to this flag output new tenuring threshold becomes equal to 1 in few minutes after application starts, even without significant load.
• We took a memory dump when the application consumed almsot all of allowed memory and according to the MAT analyzer statistics almost all of objects inn dump are unreachable(So, I can't understand why they get promoted into Old Gen)
• Survivor spaces are almost not used. Looks like new objects get promoted into Old gen right from Eden space.
• Minor GC comes each ~10-20 seconds.
• Increasing Eden/Survivor spaces didn't help to solve the problem(tenuring threshold is still 1)

So, could you please advice:

• For what reasons tenuring threshold becomes 1 so quickly?[Resolved]
• What additional steps could I do to avoid promoting new objects into old gen?

Please let me know if you need some additional info.

Edit:

JVM parameters(Updated):

-XX:MaxPermSize=512m -Xmx7g -Xmn4g -verbose:gc -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintTenuringDistribution

Application logs:

Desired survivor size 520617984 bytes, new threshold 15 (max 15)
 [PSYoungGen: 3235245K->231763K(3665920K)] 3942774K->939308K(4760064K), 0.0905430 secs] [Times: user=0.31 sys=0.00, real=0.09 secs] 
603.561: [GC
Desired survivor size 521142272 bytes, new threshold 15 (max 15)
 [PSYoungGen: 3369299K->330881K(3684864K)] 4076844K->1038442K(4779008K), 0.1343473 secs] [Times: user=0.51 sys=0.00, real=0.13 secs] 
606.347: [GC
Desired survivor size 506462208 bytes, new threshold 15 (max 15)
 [PSYoungGen: 3507329K->215655K(3685376K)] 4214890K->923233K(4779520K), 0.0925060 secs] [Times: user=0.36 sys=0.00, real=0.09 secs] 
609.084: [GC
Desired survivor size 492306432 bytes, new threshold 15 (max 15)
 [PSYoungGen: 3392103K->213344K(3713536K)] 4099681K->920945K(4807680K), 0.0802729 secs] [Times: user=0.30 sys=0.00, real=0.08 secs] 

Important detail:

Even when I start performance test with only 1 thread(that consumes ~10 MB per request) Old Gen keeps growing:

If I start GC it successfully cleanups memory

If I make a heap dump Mat Analyzer (or Yourkit) shows again that 80% of objects are unreachable

Answer 1

If your tenuring threshold drops to 1 quickly it sounds like there is some other parameter you are using which is causing this. Apart from the logging options and the maximum heap, I would try removing all other tuning arguments and only add in each one at a time when you can determine they really help. The more you set the more likely you which get some strange behavior. You could try to force the tenuring threshold but I suspect this would only hide you problem at best.

Desired survivor size 341835776 bytes, new threshold 1 (max 15)

In an attempt to increase the survivor size to avoid a full survivor triggering a a full GC, the survivor size or just one generation is using the whole young space.

I would try making the Young generation much larger. You need an Eden of at least you survivor space so your young should be at least 1 GB ideally several times larger. I would try 4 GB young or 2 Gb depending on how much space you have. e.g.

-Xmn2g -Xmx5g

or

-Xmn4g -Xmx7g

if you have plenty of memory e.g. 64 GB or more, I would try it much larger like

 -Xmn24g -Xmx32g

and reduce the young space if it doesn't appear to help.