Memory consumption issues of a Java program

Question

I have a Java program that runs on my Ubuntu 10.04 machine and, without any user interaction, repeatedly queries a MySQL database and then constructs img- and txt-files according to the data read from the DB. It makes tens of thousands of queries and creates tens of thousands of files.

After some hours of running, the available memory on my machine including swap space is totally used up. I haven't started other programs and the processes running in the background don't consume much memory and don't really grow in consumption.

To find out what is allocating so much memory I wanted to analyse a heap dump, so I started the process with -Xms64m -Xmx128m -XX:+HeapDumpOnOutOfMemoryError.

To my surprise, the situation was the same as before, after some hours the program was allocating all of the swap which is way beyond the given max of 128m.

Another run debugged with VisualVM showed that the heap allocation never is beyond the max of 128m - when the allocated memory is approximating the max, a big part of it is released again (I assume by the garbage collector).

So, it cannot be a problem a steadily growing heap.

When the memory is all used up:

free shows the following:

             total       used       free     shared    buffers     cached
Mem:       2060180    2004860      55320          0        848    1042908
-/+ buffers/cache:     961104    1099076
Swap:      3227640    3227640          0

top shows the following:

USER    VIRT    RES     SHR     COMMAND
[my_id] 504m    171m    4520    java
[my_id] 371m    162m    4368    java

(by far the two "biggest" processes and the only java processes running)

My first question is:

• How can I find out on the OS level (e.g. with command line tools) what is allocating so much memory? top / htop hasn't helped me. In case of many, many tiny processes of the same type eating up the memory: is there a way to intelligently sum up similar processes? (I know that is probably off topic as it is a Linux/Ubuntu question, but my main problem may still be Java-related)

My old questions were:

• Why isn't the memory consumption of my program given in the top output?
• How can I find out what is allocating so much memory?
• If the heap isn't the problem, is the only "allocating factor" the stack? (the stack shouldn't be a problem as there is no deep "method call depth")
• What about external resources as DB connections?

Answer 1

If indeed your Java process is the one which takes memory and there is nothing suspucios in VisualVM or memory dump then it must be somewhere in native code - either in JVM or in some of the libraries you're using. On JVM level it could be, for example, if you're using NIO or memory mapped files. If some of your libraries are using native calls or you're using not type 4 JDBC driver for your database then leak could be there.

Some suggestions:

• There are some details how to find memory leaks in native code here. Good read also.
• As usual, make sure you're properly closing all resources (Files, Streams, Connections, Threads etc). Most of these are calling native implementation at some point so memory consumed might not be directly visible in JVM
• Check resources consumed on OS level - number of open files, file descriptors, network connections etc.

Answer 2

@maximdim's answer is great general advice for this kind of situation. What is likely happening here is that a very small Java object is being retained that causes some larger amount of native(OS-level) memory to hang around. This native memory is not accounted for in the Java heap. The Java object is likely so small that you will hit your system memory limit well before the Java object retention would overwhelm the heap.

So the trick for finding this is to use successive heap dumps, far enough apart that you have noticed memory growth for the whole process, but not so far apart that a ton of work has gone on. What you are looking for are Java object counts in the heap that keep increasing and have native memory attached.

These could be file handles, sockets, db connections, or image handles just to name a few that are likely directly applicable for you.

On more rare occasions, there is a native resource that is leaked by the java implementation, even after the Java object is garbage collected. I once ran into a WinCE 5 bug where 4k were leaked with each socket close. So there was no Java object growth, but there was process memory usage growth. In these cases, it is helpful to make some counters and keep track of java allocations of objects with native memory vs. the actual growth. Then over a short enough window, you can look for any correlations and use these to make smaller testcases.

One other hint, make sure all your close operations are in finally blocks, just in case an exception is popping you out of your normal control flow. This has been known to cause this sort of problem as well.