Memory leaking without objects growing in number or size

Question

On an IBM iSeries system, I have a Java program running - an application server with a web server component, all in-house developed. When running on the 32 bit or 64 bit J9 JVM (IBM Technology for Java) I have symptoms of a memory leak.

Note that no problems are seen running this software on the iSeries classic JVM, on multiple Sun/Oracle JVMs and on Linux JVMs. Heck, I routinely leave the identical software running for weeks at a time on my wife's entry-level laptop while I am working on my website - I can assure you if it was leaking memory it would be noticed on that thing.

If I just leave a plain-vanilla system running idle, with no applications configured (basically just the messaging system and a web server), the heap just continues to slowly grow, causing more memory to be allocated over time, with each GC cycle not quite collecting down to the previous level. The pattern is exactly the same for JVMs where there is no problem, except that on those the GC sweep always reduces the heap to its previous GC level.

But, if I pull a JVM system dump at startup after stabilizing and subsequent dumps after the allocated heap has grown significantly, differential comparison indicates that the are no more reachable objects after running for a week than there were at startup. The most recent one, after a week show 6 additional classes loaded and a few objects clearly related to that. Thorough reviews of all the live objects have shown nothing which leaps out at me as unexpected.

I have tried the optimized-for-throughput and the generational-concurrent garbage collectors.

So according to the job's heap size, we appear to be leaking, and according to heap dumps, nothing is leaking.

There are no JNI methods being invoked (other than native code running as part of the core JVM), and it's definitely the heap which is growing - I can see that clearly in the IBM WRKJVMJOB information as well as reported using JMX beans in my console log file.

I cannot, so far, connect to the active JVM using JMX tools like JVisualVM because, although the listen socket is created when properly configured, the connection is rejected, apparently at a protocol level (the TCP/IP stack shows an accepted connection, but the JVM bounces it).

I am confounded, and at a loss as to where to go next.

EDIT: Just to clarify; these results are all with an uninstrumented JVM because I cannot get JMX access to this JVM (we are working on that with IBM).

EDIT 2011-11-16 19:27: I was able to pull a GC activity report over 1823 GC cycles which includes specific counts for Soft/Weak/PhantomReference counts; there is no sign of runaway growth in those numbers. There is, however significant growth in the small object tenured space (the large object tenured space is empty). It's grown from 9M to 36M.

Answer 1

Having eliminated some careless memory waste of memory (though not any leaks) in my program, and tuned the GC better for our workload, I have brought down the runaway memory use to a tolerable level.

However, in the process I have demonstrated that the IBM J9 JVM used on the AS/400 (aka iSeries, Systemi, i5, et al) has an 1336 byte/minute leak totaling 2 MB/day. I can observe this leak with a variety of programs from a "one-line" test program all the way up to our application server.

The one-line test program is this:

public class ZMemoryLeak2 extends Object {  static public synchronized void main(String... args) {     try { ZMemoryLeak2.class.wait(0); } catch(InterruptedException thr) { System.exit(0); }     }  } 

And a separate test program that did nothing except monitor memory use via the JMX API showed conclusively that 1336 B is leaked at exactly 1 minute intervals, never to be reclaimed (well, not reclaimed after 2 weeks of running). OP Note: It was actually slightly different amounts on each variation of the JVM.

Update 2012-04-02: This was accepted as a bug by IBM a few weeks ago; it was actually found and patched in Java 5 about the middle of last year, and the patch for Java 6 is expected to be available in the next week or two.