Java JAR memory usage VS class file memory usage

Question

I recently changed my large Java application to be delivered in JARs instead of individual class files. I have 405 JARS which hold 5000 class files. My problem is that when I run my program(s) as JARs (classpath is a wildcard to get all JARs) Java will continually use more and more memory. I have seen the memory go > 2GB and it seems like Java is not doing stop-the-world garbage collections to keep the memory lower. If I run the exact same program against the exploded JARs (only class files), Java's memory usage stays much lower (< 256MB) and stays there. This is happening in Oracle's Java 8 on Windows 7 (x64) and Windows Server (x64). Why would packaging my application as JARs change the memory profile? Also I have run the program for a long time as JARs with the memory maximum limited to 128MB with no problems so I don't have a memory leak.

With JAR files in classpath

With class files in classpath

Edit: I accepted the answer from @K Erlandsson because I think it is the best explanation and this is just an ugly quirk of Java. Thanks every one (and especially @K Erlandsson) for your help.

Answer 1

The first thing to note is that how much memory that is totally used on the heap is not very interesting at all times, since much of the used memory can be garbage and will be cleared by the next GC.

It is how much heap that is used by live objects that you need to be concerned about. You write in a comment:

I don't know if this matters, but if I use jvisualvm.exe to force a GC (mark sweep) the heap memory usage will drop clearing almost all the heap memory.

This matters. A lot. This means that when you see a higher heap usage when you use your jars, you see more garbage, not more memory consumed by live objects. The garbage is cleared when you do a GC and all is well.

Loading classes from jar files will consume more memory, temporarily, than loading them from class files. The jar files need to be opened, seeked, and read from. This requires more operations and more temporary data than simply opening a specific .class file and reading it.

Since most of the heap usage is cleared by a GC, this additional memory consumption is not something you need to be very concerned about.

You also write:

Java will continually use more and more memory. I have seen the memory go > 2GB and it seems like Java is not doing stop-the-world garbage collections to keep the memory lower.

This is typical behavior. The GC only runs when the JVM thinks it is necessary. The JVM will tune this depending on memory behavior.

Edit: Now that we see your jConsole images we see a difference in committed heap memory (250 mb vs 680 mb). Committed heap is the actual size of the heap. This will vary (up to what you set with -Xmx) depending on what that JVM thinks will yield the best performance for your application. However, it will mostly increase, almost never decrease.

For the jar case the JVM has assigned a bigger heap to your application. Probably due to more memory being required during the initial class loading. The JVM then thought a bigger heap would be faster.

When you have a bigger heap, more committed memory, there is more memory to use before running a GC. That is why you see the difference in memory usage in the two case.

Bottom line: All the extra usage you see is garbage, not live objets, why you do not need to be concerned about this behavior unless you have an actual problem since the memory will be reclaimed on the next GC.