How can one `finalize` invocation break GC / JVM?

Question

While investigating Why ThreadPoolExecutor behaves differently when running Java program in Eclipse and from command line? I wrote a test that throws a very strange OutOfMemoryError (max mem = 256 Mb)

class A {
    byte[] buf = new byte[150_000_000];

    protected void finalize() {
        int i = 1;
    }
}

A a1 = new A();
a1 = null;
A a2 = new A();

comment out int i = 1 and the test works. As far as I understand when finalize is empty HotSpot simply ignores it. But how can just one practically empty finalize invocation break GC / JVM?

Answer 1

But how can just one empty finalize invocation break GC / JVM?

When there's a finalizer, objects survive one more round of garbage collection than they would otherwise (as the object itself has to be kept alive until it's finalized). Therefore if you have a large object with a finalizer, that will naturally lead to an OutOfMemoryError occurring in situations when it wouldn't without a finalizer.

In this code:

A a1 = new A();
a1 = null;
A a2 = new A();

... the GC will trigger on the last line in order to try to find enough memory to allocate the second A. Unfortunately, it can't garbage collect the first A (and the array it refers to) because the finalizer hasn't run yet. It doesn't wait until the finalizer completes, then try to garbage collect again - it just throws OutOfMemoryError.