what will the Finalizer thread do if there is a infinite loop or deadlock in the Java finalize method

Question

what will the Finalizer thread do if there is a infinite loop or deadlock in the Java finalize method.

Answer 1

The spec writes:

Before the storage for an object is reclaimed by the garbage collector, the Java Virtual Machine will invoke the finalizer of that object.

The Java programming language does not specify how soon a finalizer will be invoked, except to say that it will happen before the storage for the object is reused.

I read this to mean that the finalizer must have completed before the storage may be reused.

The Java programming language does not specify which thread will invoke the finalizer for any given object.

It is important to note that many finalizer threads may be active (this is sometimes needed on large shared memory multiprocessors), and that if a large connected data structure becomes garbage, all of the finalize methods for every object in that data structure could be invoked at the same time, each finalizer invocation running in a different thread.

That is, finalization may occur in the garbage collector thread, in a separate thead, or even a separate thread pool.

A JVM is not permitted to simply abort executing a finalizer, and can only use a finite number of threads (threads are operating system resources, and operating systems don't support arbitrarily many threads). Non-terminating finalizers will therefore of necessity starve that thread pool, thereby inhibit collection of any finalizable objects, and cause a memory leak.

The following test program confirms this behavior:

public class Test {

    byte[] memoryHog = new byte[1024 * 1024];

    @Override
    protected void finalize() throws Throwable {
        System.out.println("Finalizing " + this + " in thread " + Thread.currentThread());
        for (;;);
    }

    public static void main(String[] args) {
        for (int i = 0; i < 1000; i++) {
            new Test();
        }
    }
}

On Oracle JDK 7, this prints:

Finalizing tools.Test@1f1fba0 in thread Thread[Finalizer,8,system]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at tools.Test.<init>(Test.java:5)
        at tools.Test.main(Test.java:15)

Answer 2

I would say that since the Java Specification doesn't tell how the finalize method must be invoked (just that it must be invoked, before the object is garbage collected), the behaviour is implementation specific.

The spec doesn't rule out having multiple threads running the process, but doesn't require it:

It is important to note that many finalizer threads may be active (this is sometimes needed on large shared memory multiprocessors), and that if a large connected data structure becomes garbage, all of the finalize methods for every object in that data structure could be invoked at the same time, each finalizer invocation running in a different thread.

Looking at the sources of the JDK7, the FinalizerThread keeps the queue of objects scheduled for finalization (actually objects are added to the queue by the GC, when proven to be unreachable - check ReferenceQueue doc):

private static class FinalizerThread extends Thread {
    private volatile boolean running;
    FinalizerThread(ThreadGroup g) {
        super(g, "Finalizer");
    }
    public void run() {
        if (running)
            return;
        running = true;
        for (;;) {
            try {
                Finalizer f = (Finalizer)queue.remove();
                f.runFinalizer();
            } catch (InterruptedException x) {
                continue;
            }
        }
    }
}

Each object is removed from the queue, and runFinalizer method is run on it. Check is done if the finalization had run on the object, and if not it is being invoked, as a call to a native method invokeFinalizeMethod. The method simply is calling the finalize method on the object:

JNIEXPORT void JNICALL
Java_java_lang_ref_Finalizer_invokeFinalizeMethod(JNIEnv *env, jclass clazz,
                                                  jobject ob)
{
    jclass cls;
    jmethodID mid;

    cls = (*env)->GetObjectClass(env, ob);
    if (cls == NULL) return;
    mid = (*env)->GetMethodID(env, cls, "finalize", "()V");
    if (mid == NULL) return;
    (*env)->CallVoidMethod(env, ob, mid);
}

This should lead to a situation, where the objects get queued in the list, while the FinalizerThread is blocked on the faulty object, which in turn should lead to OutOfMemoryError.

So to answer the original question:

what will the Finalizer thread do if there is a infinite loop or deadlock in the Java finalize method.

It will simply sit there and run that infinite loop until OutOfMemoryError.

public class FinalizeLoop {
    public static void main(String[] args) {
        Thread thread = new Thread() {
            @Override
            public void run() {
                for (;;) {
                    new FinalizeLoop();
                }
            }
        };
        thread.setDaemon(true);
        thread.start();
        while (true);
    }

    @Override
    protected void finalize() throws Throwable {
        super.finalize();
        System.out.println("Finalize called");
        while (true);

    }
}

Note the "Finalize called" if printed only once on the JDK6 and JDK7.