java.lang.OutOfMemoryError: Compressed class space

Question

We are running on java-8-oracle.

We moved to java8 six month ago.

In the past few days we have been getting an OOME from time to time and we haven't been able to identify or reproduce the problem.

When we execute a call to the server (tomcat) we get this error on the stacktrace:

java.lang.OutOfMemoryError: Compressed class space

Restarting the server solves the problem. The same call to other server works, and so does another call of another type to the same server.

When looking on the gc.log we see:

2015-05-27T16:05:42.991+0000: 98774.440: [Full GC (Last ditch collection) 98774.440: [CMS: 575745K->575330K(3495936K), 0.8687777 secs] 575745K->575330K(4107008K), [Metaspace: 97940K->97940K(1396736K)], 0.8696093 secs] [Times: user=0.95 sys=0.00, real=0.88 secs]
2015-05-27T16:05:55.486+0000: 98786.935: [Full GC (Metadata GC Threshold) 98786.935: [CMS: 573414K->578735K(3495936K), 0.9372859 secs] 925046K->578735K(4107008K), [Metaspace: 99428K->99428K(1396736K)], 0.9386626 secs] [Times: user=1.01 sys=0.00, real=0.94 secs]

jstat -gc returns:

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
87296.0 87296.0  0.0   3151.4 523776.0 148284.4 3495936.0   574868.5  1395640.0 98066.3 1048576.0 11339.1  12165  636.851  223   116.957  

753.808

I don't see any memory problems either in the jstat log or in the gc log.

Trying to run jmap -clstats hangs:

Attaching to process ID 5110, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.25-b02
finding class loader instances ..

Answer 1

We faced a similar issue. Unfortunately heapdumps won't help you since the classes are not in the heap but in native memory. Enable these in your JVM settings to troubleshoot the classes loaded:

-XX:+PrintGCDetails -XX:+TraceClassUnloading -XX:+TraceClassLoading

In our case the issue was JAXBContext.newInstance not being singleton.

Good luck, Albert

Answer 2

With compressed oops and compressed class pointers the available space for classes is constrained due to the necessary pointer mangling. 1GB in your case.

That's a lot of classes, so this might be an indicate that something in your application is creating a lot of classes and never releasing them. Application reload maybe?

If you are certain that your application just needs that much memory for classes you can try bumping the limit via -XX:CompressedClassSpaceSize=... or disabling compressed class pointers via -XX:-UseCompressedClassPointers.

Note that by default compressed class space + compressed heap (+ some overhead) cannot exceed 32GB. Although, AIUI, changing object alignment can bump that limit further.

Otherwise you should take a heapdump and analyze what's holding onto the loaded classes.