What did I do that made "weak refs processing" take 30 sec instead of 1.5 sec?

Question

The story

My server is running with 24x2 processors, and the java heap is about 70 GB. At some point after installing a new version (version-B), I saw that Full GC is taking around 30sec (stopping all threads). After enabling the XX:+ParallelRefProcEnabled the weak ref processing went down to around 3-6 sec. But this is just a "Bend aid", not a cure.

2011-03-22T20:38:24.276+0000: 29540.794: [GC[YG occupancy: 5477281 K (7549760 K)]29540.794: [Rescan (parallel) , 0.4083780 secs]29541.203: [weak refs processing, 3.2855240 secs]29544.488: [class unloading, 0.0187270 secs]29544.507: [scrub symbol & string tables, 0.0095530 secs] [1 CMS-remark: 102801236K(114294784K)] 108278518K(121844544K), 3.7319690 secs] [Times: user=65.53 sys=0.14, real=3.73 secs]

Before version-B, (and without the ParallelRefProcEnabled flag) the weak ref processing used to take around 1.5 sec. (For roughly the same load)

The purpose:

What I'm trying to find out is, what was the change introduced version-B that caused the processing to jump from 1.5 sec to 30 sec. There are a few changes in version-B, and no real suspect involving weak-references.

I want to change my code so it would not relay on heavy weak ref processing.

Questions:

• I would like to understand what exactly is happening in the "weak ref processing" stage, so I can look for a suspect, or justify a code rewrite. Any good resource for reading about what exactly happens in that stage?

• What are the possible causes for long "weak refs processing"? (Number of weak references instances, number of objects holding a weak-reference, depth of reference-tree which is weakly referenced,...)

More info:

• The CPU usage is not so high and doesn't seem to be an issue
• The GC (that includes weak reference processing) occurs about every 8 minutes.
• Running Java Sun, 1.6.0_20

I'd appreciate any response, Thanks, Erez.

Answer 1

AFAIK, The Reference processing time is the time it takes to determine whether a reference can be collected or not. This should be propoprtional to the number of WeakReferences you have.

It may be that a change in data structure is possible which reduces the number of weak references dramatically. e.g. say you have a cache using Map of WeakReferences. If this can be replaced with a WeakReference to a Map you may get the same outcome but with dramatically less references.

You may see a performance improvement with Java 6 update 24 which has a much newer JVM (about a year newer)