We have a java process running on Solaris 10 serving about 200-300 concurrent users. The administrators have reported that memory used by process increases significantly over time. It reaches 2GB in few days and never stops growing.
We have dumped the heap and analysed it using Eclipse Memory Profiler, but weren't able to see anything out of the ordinary there. The heap size was very small.
After adding memory stat logging, to our application we have found discrepancy between memory usage reported by "top" utility, used by the administrator, and the usage reported by MemoryMXBean and Runtime libraries.
Here is an output from both.
Memory usage information
From the Runtime library
Free memory: 381MB
Allocated memory: 74MB
Max memory: 456MB
Total free memory: 381MB
From the MemoryMXBean library.
Heap Committed: 136MB
Heap Init: 64MB
Heap Used: 74MB
Heap Max: 456MB
Non Heap Committed: 73MB
Non Heap Init: 4MB
Non Heap Used: 72MB
Current idle threads: 4
Current total threads: 13
Current busy threads: 9
Current queue size: 0
Max threads: 200
Min threads: 8
Idle Timeout: 60000
PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND
99802 axuser 115 59 0 2037M 1471M sleep 503:46 0.14% java
How can this be? top command reports so much more usage. I was expecting that RES should be close to heap+non-heap.
pmap -x , however, reports most of the memory in the heap:
Address Kbytes RSS Anon Locked Mode Mapped File
*102000 56 56 56 - rwx---- [ heap ]
*110000 3008 3008 2752 - rwx---- [ heap ]
*400000 1622016 1621056 1167568 - rwx---- [ heap ]
*000000 45056 45056 45056 - rw----- [ anon ]
Can anyone please shed some light on this? I'm completely lost.
Thanks.
Update
This does not appear to be an issue on Linux.
Also, based on the Peter Lawrey's response the "heap" reported by pmap is native heap not Java heap.
Use reference objects to avoid memory leaks Using the java. lang. ref package, you can work with the garbage collector in your program. This allows you to avoid directly referencing objects and use special reference objects that the garbage collector easily clears.
A memory leak is bad because it blocks memory resources and degrades system performance over time. If not dealt with, the application will eventually exhaust its resources, finally terminating with a fatal java.
Java is also a very high-level Object-Oriented programming language (OOP) which means that while the application code itself is much easier to maintain, the objects that are instantiated will use that much more memory.
I have encountered a similar problem and found a resolution:
Solaris 11
JDK10
REST application using HTTPS (jetty server)
There was a significant increase of c-heap (observed via pmap) over time
I decided to do some stress tests with libumem. So i started the proces with
UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1
and stressed the application with https requests. After a while I connected to the process with mdb. In mdb I used the command ::findleaks and it showed this as a leak:
libucrypto.so.1`ucrypto_digest_init
So it seems than the JCA (Java Cryptography Architecture) implementation OracleUcrypto has some issues on Solaris.
The problem was resolved by updating of the $JAVA_HOME/conf/security/java.security file - I changed the priority of OracleUcrypto to 3 and the SUN implementation to 1
security.provider.3=OracleUcrypto
security.provider.2=SunPKCS11 ${java.home}/conf/security/sunpkcs11-solaris.cfg
security.provider.1=SUN
After this the problem dissapeared.
This also explains why there is no problem on linux - since there are different implememntations of JCA providers in play
In garbage collected environments, holding on to unused pointers amounts to "failure to leak" and prevents the GC from doing its job. It's really easy to accidentally keep pointers around.
A common culprit is hashtables. Another is arrays or vectors which are logically cleared (by setting the reuse index to 0) but where the actual contents of the array (above the use index) is still pointing to something.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With