Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Java process memory growing indefinitely. Memory leak?

We have a java process running on Solaris 10 serving about 200-300 concurrent users. The administrators have reported that memory used by process increases significantly over time. It reaches 2GB in few days and never stops growing.

We have dumped the heap and analysed it using Eclipse Memory Profiler, but weren't able to see anything out of the ordinary there. The heap size was very small.

After adding memory stat logging, to our application we have found discrepancy between memory usage reported by "top" utility, used by the administrator, and the usage reported by MemoryMXBean and Runtime libraries.

Here is an output from both.

Memory usage information 

From the Runtime library
Free memory: 381MB
Allocated memory: 74MB
Max memory: 456MB
Total free memory: 381MB

From the MemoryMXBean library.
Heap Committed: 136MB
Heap Init: 64MB
Heap Used: 74MB
Heap Max: 456MB
Non Heap Committed: 73MB
Non Heap Init: 4MB
Non Heap Used: 72MB

Current idle threads: 4
Current total threads: 13
Current busy threads: 9
Current queue size: 0
Max threads: 200
Min threads: 8
Idle Timeout: 60000

  PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME   CPU COMMAND
99802 axuser   115   59    0 2037M 1471M sleep  503:46 0.14% java

How can this be? top command reports so much more usage. I was expecting that RES should be close to heap+non-heap.

pmap -x , however, reports most of the memory in the heap:

Address     Kbytes       RSS       Anon     Locked Mode   Mapped File
*102000         56         56         56       - rwx----    [ heap ]
*110000       3008       3008       2752       - rwx----    [ heap ]
*400000    1622016    1621056    1167568       - rwx----    [ heap ]
*000000      45056      45056      45056       - rw-----    [ anon ]

Can anyone please shed some light on this? I'm completely lost.

Thanks.

Update

This does not appear to be an issue on Linux.

Also, based on the Peter Lawrey's response the "heap" reported by pmap is native heap not Java heap.

like image 672
Srki Rakic Avatar asked Mar 23 '16 18:03

Srki Rakic


People also ask

How do you fix a memory leak in Java?

Use reference objects to avoid memory leaks Using the java. lang. ref package, you can work with the garbage collector in your program. This allows you to avoid directly referencing objects and use special reference objects that the garbage collector easily clears.

Are memory leaks a problem in Java?

A memory leak is bad because it blocks memory resources and degrades system performance over time. If not dealt with, the application will eventually exhaust its resources, finally terminating with a fatal java.

Why is my Java taking up so much memory?

Java is also a very high-level Object-Oriented programming language (OOP) which means that while the application code itself is much easier to maintain, the objects that are instantiated will use that much more memory.


2 Answers

I have encountered a similar problem and found a resolution:

Solaris 11
JDK10
REST application using HTTPS (jetty server)
There was a significant increase of c-heap (observed via pmap) over time

I decided to do some stress tests with libumem. So i started the proces with

UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1

and stressed the application with https requests. After a while I connected to the process with mdb. In mdb I used the command ::findleaks and it showed this as a leak:

libucrypto.so.1`ucrypto_digest_init

So it seems than the JCA (Java Cryptography Architecture) implementation OracleUcrypto has some issues on Solaris.

The problem was resolved by updating of the $JAVA_HOME/conf/security/java.security file - I changed the priority of OracleUcrypto to 3 and the SUN implementation to 1

security.provider.3=OracleUcrypto
security.provider.2=SunPKCS11 ${java.home}/conf/security/sunpkcs11-solaris.cfg
security.provider.1=SUN

After this the problem dissapeared.

This also explains why there is no problem on linux - since there are different implememntations of JCA providers in play

like image 119
Deniso Avatar answered Sep 22 '22 11:09

Deniso


In garbage collected environments, holding on to unused pointers amounts to "failure to leak" and prevents the GC from doing its job. It's really easy to accidentally keep pointers around.

A common culprit is hashtables. Another is arrays or vectors which are logically cleared (by setting the reuse index to 0) but where the actual contents of the array (above the use index) is still pointing to something.

like image 45
ddyer Avatar answered Sep 19 '22 11:09

ddyer