Menu
I have 7 different java daemons that I run (all 7) on 3 different servers. The java command line has -Xmx2048m and -Xss1024k. On these 3 servers, all 21 processes show just under 2.5 GB for VIRT size in top and atop. RES size varies from 300 to 1.9 GB according to which daemon it is.
That is all as it should be.
Enter the new server. Faster CPU, more RAM (16 GB instead of 8 GB), slightly newer java (1.6.0_10-b33 on the old servers, 1.6.0_31-b04 on the new server). Both systems (and JVMs) are 64bit.
Moved 2 of the daemons to the new server. On the new server, given the same task, the daemons are both consuming vastly more CPU (about a core's worth) and getting Less done. (Moved from 5110 processors on the old systems to 5620s on the new one).
Pretty much a full extra core of CPU usage (GC thread??) and reporting 5 GB VIRT and 2 GB RES for one daemon and 10.5 GB VIRT and 2 GB RES for the other daemon.
Any ideas what would cause java to ignore (or appear to ignore if that is the case) the memory limits?
299
As you might recall, Java is a garbage-collected language. In order for the garbage collector to know which objects are eligible for collection, it needs to keep track of the object graphs. So this is one part of the memory lost to this internal bookkeeping.
The default values for Xmx is based on the physical memory of the machine. The Xmx value is 25% of the available memory with a maximum of 25 GB. However, where there is 2 GB or less of physical memory, the value set is 50% of available memory with a minimum value of 16 MB and a maximum value of 512 MB.
-XMX indicates the maximum memory allocated during JVM running. For example, -Xms500m indicates that the JVM process can occupy a maximum of 500 MB memory. 3. -XSS Memory size allocated to each thread started by the JVM. The default value is 256 KB in JDK1.
High heap usage occurs when the garbage collection process cannot keep up. An indicator of high heap usage is when the garbage collection is incapable of reducing the heap usage to around 30%. In the image above you can see normal sawtooth of JVM heap.
Turns out this is a glibc problem.
The short answer for me was:
export MALLOC_ARENA_MAX=1
This decreased process footprint (VIRT in top) by as much as 5x. Back to the levels seen in CentOS 5.
Recent versions of glibc have a new feature "per-thread memory pools":
http://www.centos.org/docs/5/html/5.4/Technical_Notes/glibc.html
The last item in the 1.71.1 log section discusses it (and refers to a non-public bug....)
166
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
