java OOM on creating 2 arrays of one billion ints

Question

I am writing simple program in java to create 2 int arrays of 1 billion size. I ran this program with -Xms10G, i.e. 10GB of memory still I got OOM error. Below is the snippet.

public class TestBigIntArraySize {
  public static int arraySize = 1000_000_000;
  public static int [] firstArray = new int[arraySize];
  public static int [] secondArray = new int[arraySize];

  public static void main(String[] args) {
    System.out.println(1000_000_000 * Integer.SIZE);
  }
}

As far as I can think the memory used for 1 billion int array would be System.out.println(1000_000_000 * Integer.SIZE); which returns 1,935,228,928 which is lesser than 2GB. So my programs total requirements would be max 4GB.

I get error even if I create arrays in a method call and return array or static (like below) or inside main(). The memory that is required for it work is 12G which is 3 times of what I expected. I am using oracle java : jdk1.8.0_201

I tried the option -Xms10G -XX:NewRatio=1 --- which Worked.

But I want to decrease my memory footprint further.

I tried option to giving more memory to eden by -Xms9G -XX:NewRatio=0.5 but java complains illegal argument.

I tried option to directly allocate array to old gen by -Xms9G -XX:NewRatio=1 -XX:PretenureSizeThreshold=10000. But this also give OOM.

It's just an experimental project and I am just manipulating the array's in place. I would like to do it in smallest possible memory. Can someone suggest how to go about it? What java options and why?

Answer 1

So let's say multiple by bit length is probably not the best way to get a byte count. As @mayamar has mentioned, your actual memory usage is around 2*4 gigabyte.

Anyway, let's get to the actual tuning phrase. 4GB probably too large and will be stored in old gen directly. So you need to increase old gen size. Changing new gen settings may work but that's... well, you are effectively turning off the old gen then. It may do harm to your other part of the test cases.

Your attempt NewRatio=1 make new to old gen ratio being 1:1, instead of something better, e.g. 1:100. Yet, if you make the ratio too great JVM may not be able to boot (GC during VM initialization). It's better to just specify it with MaxNewSize.

In the end, running something similiar to this one will be very close to your "minimize memory footprint" requirement.

java -Xmx8400000000 -XX:MaxNewSize=30M -XX:OldSize=8300000000 TestBigIntArraySize 

Note: it's better to spare several dozens of megabyte as JVM itself will need memory to operate. If your program is not as small as your MVCE is, you will need to left out some more space if you don't wish GC kicks in every now and then.

Answer 2

The same problem as here.

In JDK 8, which uses Parallel GC by default, 10 GB heap is divided into 6.67 GB Old Generation + 3.33 GB Young Generation. So, there is no room to fit two contiguous blocks of 3.72 GB (1 billion four-byte integers).

The easiest way to solve the problem is to turn on G1 GC and avoid tricky generation sizing at all. Your example will then work with 8 GB heap:

java -XX:+UseG1GC -Xmx8g TestBigIntArraySize