What happens when the JVM runs out of memory to allocate during run time?

Question

After thinking for a long time of a generic way to pose this question (and failing to find one) I'm just going to ask it as a concrete example:

Suppose I have a Linux machine which has 1 Gb of memory which it can allocate to processes (physical and swap totals 1 Gb).

I have a standard Oracle Hotspot JVM version 7 installed on the machine. If at a given moment, there are enough programs running such that only 400 Mb of that 1 Gb are free, and I start a Java program at that moment with the following JVM flags:

java -Xms256m -Xmx512m -jar myJar.jar

what happends? :

A. does the JVM fail to start right away because it will try to allocate all of the 512 Mb of memory and fail (due to the fact that there's not enough available memory at the moment)?

if the JVM starts:

if at some point the running Java process will need more than 400 Mb of memory (and there's still only 400 Mb of memory that's free other than what the current Java process has already used), what will happen:

B. will the Java process fail with an OutOfMemroyError?

C. will it fail with some other (standard) error?

D. is it undefined behavior?

Answer 1

-Xmx just defines the maximum size of the heap. It makes no guarantee on wether there is so much memory or not. It only ensures that the heap will never be bigger then the given value. That said, Option B.) will happen, an outOfMemoryError will be thrown.