What happens when there's insufficient memory to throw an OutOfMemoryError?

Question

I am aware that every object requires heap memory and every primitive/reference on the stack requires stack memory.

When I attempt to create an object on the heap and there's insufficient memory to do so, the JVM creates an java.lang.OutOfMemoryError on the heap and throws it to me.

So implicitly, this means that there is some memory reserved by the JVM on startup.

What happens when this reserved memory is used up (it would definitely be used up, read discussion below) and the JVM does not have enough memory on the heap to create an instance of java.lang.OutOfMemoryError?

Does it just hang? Or would he throw me a null since there's no memory to new an instance of OOM ?

try {
    Object o = new Object();
    // and operations which require memory (well.. that's like everything)
} catch (java.lang.OutOfMemoryError e) {
    // JVM had insufficient memory to create an instance of java.lang.OutOfMemoryError to throw to us
    // what next? hangs here, stuck forever?
    // or would the machine decide to throw us a "null" ? (since it doesn't have memory to throw us anything more useful than a null)
    e.printStackTrace(); // e.printStackTrace() requires memory too.. =X
}

==

Why couldn't the JVM reserve sufficient memory?

No matter how much memory is reserved, it is still possible for that memory to be used up if the JVM does not have a way to "reclaim" that memory:

try {
    Object o = new Object();
} catch (java.lang.OutOfMemoryError e) {
    // JVM had 100 units of "spare memory". 1 is used to create this OOM.
    try {
        e.printStackTrace();
    } catch (java.lang.OutOfMemoryError e2) {
        // JVM had 99 units of "spare memory". 1 is used to create this OOM.
        try {
            e.printStackTrace();
        } catch (java.lang.OutOfMemoryError e3) {
            // JVM had 98 units of "spare memory". 1 is used to create this OOM.
            try {
                e.printStackTrace();
            } catch (java.lang.OutOfMemoryError e4) {
                // JVM had 97 units of "spare memory". 1 is used to create this OOM.
                try {
                    e.printStackTrace();
                } catch (java.lang.OutOfMemoryError e5) {
                    // JVM had 96 units of "spare memory". 1 is used to create this OOM.
                    try {
                        e.printStackTrace();
                    } catch (java.lang.OutOfMemoryError e6) {
                        // JVM had 95 units of "spare memory". 1 is used to create this OOM.
                        e.printStackTrace();
                        //........the JVM can't have infinite reserved memory, he's going to run out in the end
                    }
                }
            }
        }
    }
}

Or more concisely:

private void OnOOM(java.lang.OutOfMemoryError e) {
    try {
        e.printStackTrace();
    } catch (java.lang.OutOfMemoryError e2) {
        OnOOM(e2);
    }
}

Answer 1

The JVM never really runs out of memory. It does memory computation of the heap stack in advance.

The Structure of the JVM, Chapter 3, section 3.5.2 states:

• If Java virtual machine stacks can be dynamically expanded, and expansion is attempted but insufficient memory can be made available to effect the expansion, or if insufficient memory can be made available to create the initial Java virtual machine stack for a new thread, the Java virtual machine throws an OutOfMemoryError.

For Heap, Section 3.5.3.

• If a computation requires more heap than can be made available by the automatic storage management system, the Java virtual machine throws an OutOfMemoryError.

So, it does a computation in advance before doing allocation of the object.

---

What happens is that the JVM tries to allocate memory for an object in the memory called Permanent Generation region (or PermSpace). If allocation fails (even after the JVM invokes the Garbage Collector to try & allocate free space), it throws an OutOfMemoryError. Even exceptions requires a memory space so the error will be thrown indefinitely.

Further reading.? Furthermore, OutOfMemoryError can occur in different JVM structure.

Answer 2

Graham Borland seems to be right: at least my JVM apparently re-uses OutOfMemoryErrors. To test this, I wrote a simple test program:

class OOMTest {
    private static void test (OutOfMemoryError o) {
        try {
            for (int n = 1; true; n += n) {
                int[] foo = new int[n];
            }
        } catch (OutOfMemoryError e) {
            if (e == o)
                System.out.println("Got the same OutOfMemoryError twice: " + e);
            else test(e);
        }
    }
    public static void main (String[] args) {
        test(null);
    }
}

Running it produces this output:

$ javac OOMTest.java && java -Xmx10m OOMTest 
Got the same OutOfMemoryError twice: java.lang.OutOfMemoryError: Java heap space

BTW, the JVM I'm running (on Ubuntu 10.04) is this:

$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

---

Edit: I tried to see what would happen if I forced the JVM to run completely out of memory using the following program:

class OOMTest2 {
    private static void test (int n) {
        int[] foo;
        try {
            foo = new int[n];
            test(n * 2);
        }
        catch (OutOfMemoryError e) {
            test((n+1) / 2);
        }
    }
    public static void main (String[] args) {
        test(1);
    }
}

As it turns out, it seems to loop forever. However, curiously, trying to terminate the program with Ctrl+C doesn't work, but only gives the following message:

Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGINT to handler- the VM may need to be forcibly terminated