Create huge dictionary

Question

In my quest to the primes, I've already asked this question : Can't create huge arrays which lead me to create my own class of fake arrays based on a dictionary of arrays... : private Dictionary<int, Array> arrays = new Dictionary<int, Array>();

I can know create fake arrays of a lot of bool (like 10 000 000 000) using the code below:

public class CustomArray
{
    private Dictionary<int, Array> arrays = new Dictionary<int, Array>();

    public CustomArray(ulong lenght)
    {
        int i = 0;
        while (lenght > 0x7FFFFFC7)
        {
            lenght -= 0x7FFFFFC7;
            arrays[i] = new bool[0x7FFFFFC7];
            i++;
        }
        arrays[i] = new bool[lenght];
    }
}

But it crashes as soon as I ask for a CustomArray of 100 000 000 000 elements. It works well for the 25 first iterations (my Dictionary contains 25 arrays of 0x7FFFFFC7 elements) but then it crashes with an OutOfMemory exception.

_{As a remainder, I've got 16GB memory, VS2013, the program is compiled in 64bits, I've enabled the gcAllowVeryLargeObjects option and I don't see any memory peak in the Task Manager.}

---

How can I avoid this error?

Answer 1

100000000000 bools means ~93 GB of memory. You only have @50 GB (including the default allocated virtual memory).

Storing them as bits (not as bytes), would get you down to ~12GB.

Look at System.Collection.BitArray