Creating a large array of numbers (10^9 size)

Question

I want to create an array that is capable of storing 10^9 numbers(long int).If i try doing this my compiler crashes.What is the maximum size array allowed in C++.Also if i do this dynamically too i get the same problem.How can i accomplish the task i am looking to acheive? Thanks, any help would be appreciated.

Answer 1

The maximum array size is dependent on the data you store (and the integers available to index them).

So on a 32bit system, you can only index 2³² elements at most if you're lucky, which is a bit above 10⁹. On a 64bit system, you can index 2⁶⁴ elements, which is a bit above 10¹⁹. This is essentially the maximum array size. Being able to index that does not imply that you can also actually get that much from the operating system, as the actual virtual address space might be much smaller. On Linux, a virtual adress space of approx. 64 Terabytes is available per process on 64bit, which are 2⁴² bytes.

However, if you actually try to allocate this, you need that much amount of bytes! So if you try to allocate an array of long int which will probably be 64bits of size, you need 8 Gigabytes of memory.

On a 32bit system, this is impossible. On a 64bit system, you need to have that amount of ram and swap space to work.

If you're on a 32bit system or on a 64bit system without enough memory, you'll get a out of memory error, which is probably the reason for the behaviour you see.

If you also try to create the array statically in a .data section of your executable, the executable may end up with being 8 GBytes large, where you could run into filesystem limits (fat32 anyone?). Also the compiler probably chokes on the amount of data (on 32bit, it'll probably crash).

If you're allocating on stack (this is, as a statically sized local variable array), you'll also run into stack limits on certain operating systems.

Answer 2

An array of 10^9 longs would typically take up at least 4GB of memory, which would already be prohibitive in all 32-bit systems.

Even if that much memory is available in a 64-bit system you certainly cannot expect to allocate 4GB on the stack like this:

void foo() {
    long arr[1000000000]; // stack size is a typically few MBs!
}

Answer 3

I don't think it's your compiler crash, it's your memory crash (out of memory or something like this), e.g in windows 32 bit, at most you could use 2^32 bit of memory space which is smaller than 10^9*64 so you will get memory exception. You could use it by paging and loading smaller parts from file to memory.

Edit: As Tobias Langner mentioned in comments, compiler also could raise this error but original problem is from memory.