Odd Java StackOverflowError?

Question

Why does this code totally destroy the output?

public class Main {
    public static void main(String[] args) {
        System.out.println();

        rec();
    }

    private static int rec() {
        try {
            return rec();
        } catch(StackOverflowError e) {
            System.out.println("Boo.");
            return 0;
        }
    }
}

Sample output I get: Boo.Boo.Boo.Boo.Boo.Boo.Boo.Boo.Boo.

Answer 1

The problem is similar to the problem in this Question: Stack overflow error handling in finally block

Basically, the println call your handler for StackOverflowException is triggering a further StackOverflowException. This is being handled in the enclosing call to rec(). This continues (unwinding the rec() calls as we go) until the exception handler has enough space to complete the println call. The intermediate calls are probably adding characters to the output buffer, and then raising the SOE.

A precise explanation would require forensic analysis of the code of PrintStream and the stream stack ... all the way to the point where it calls into native code.

---

If you want a meta-explanation, it is that your code is attempting to recover from an Error ... which the javadocs say is something that you should not attempt to do. As a general rule, after an Error, the JVM is going to be an uncertain, and possibly broken state. In this case, the uncertainty manifests as data that may or may not have been written into the buffer. The behaviour is (probably) deterministic, but definitely hard to analyse and impossible to predict without a proper analysis.