A workaround for a big multidimensional array (Jagged Array) C#?

Question

I'm trying to initialize an array in three dimension to load a voxel world.

The total size of the map should be (2048/1024/2048). I tried to initialize an jagged array of "int" but I throw a memory exception. What is the size limit? Size of my table: 2048 * 1024 * 2048 = 4'191'893'824

Anyone know there a way around this problem?

// System.OutOfMemoryException here !
int[][][] matrice = CreateJaggedArray<int[][][]>(2048,1024,2048);
// if i try normal Initialization I also throws the exception
int[, ,] matrice = new int[2048,1024,2048];

    static T CreateJaggedArray<T>(params int[] lengths)
    {
        return (T)InitializeJaggedArray(typeof(T).GetElementType(), 0, lengths);
    }

    static object InitializeJaggedArray(Type type, int index, int[] lengths)
    {
        Array array = Array.CreateInstance(type, lengths[index]);
        Type elementType = type.GetElementType();

        if (elementType != null)
        {
            for (int i = 0; i < lengths[index]; i++)
            {
                array.SetValue(
                    InitializeJaggedArray(elementType, index + 1, lengths), i);
            }
        }

        return array;
    }

Answer 1

The maximum size of a single object in C# is 2GB. Since you are creating a multi-dimensional array rather than a jagged array (despite the name of your method) it is a single object that needs to contain all of those items, not several. If you actually used a jagged array then you wouldn't have a single item with all of that data (even though the total memory footprint would be a tad larger, not smaller, it's just spread out more).

Answer 2

Thank you so much to all the staff who tried to help me in understanding and solving my problem.

I tried several solution to be able to load a lot of data and stored in a table. After two days, here are my tests and finally the solution which can store 4'191'893'824 entry into one array

I add my final solution, hoping someone could help

the goal

I recall the goal: Initialize an integer array [2048/1024/2048] for storing 4'191'893'824 data

---

Test 1: with JaggedArray method (failure)

---

system out of memory exception thrown

            /* ******************** */
            /* Jagged Array method  */
            /* ******************** */
            
            // allocate the first dimension;
            bigData = new int[2048][][];
            for (int x = 0; x < 2048; x++)
            {
                // allocate the second dimension;
                bigData[x] = new int[1024][];
                for (int y = 0; y < 1024; y++)
                {
                    // the last dimension allocation
                    bigData[x][y] = new int[2048];
                }
            }

---

Test 2: with List method (failure)

---

system out of memory exception thrown (divide the big array into several small array .. Does not work because "List <>" allows a maximum of "2GB" Ram allocution like a simple array unfortunately.)

        /* ******************** */
        /* List method          */
        /* ******************** */
        
        List<int[,,]> bigData = new List<int[,,]>(512);
        for (int a = 0; a < 512; a++)
        {
            bigData.Add(new int[256, 128, 256]);
        }
   

---

Test 3: with MemoryMappedFile (Solution)

---

I finally finally found the solution! Use the class "Memory Mapped File" contains the contents of a file in virtual memory.

MemoryMappedFile MSDN Use with custom class that I found on codeproject here. The initialization is long but it works well!

        /* ************************ */
        /* MemoryMappedFile method  */
        /* ************************ */

        string path = AppDomain.CurrentDomain.BaseDirectory;            
        var myList = new GenericMemoryMappedArray<int>(2048L*1024L*2048L, path); 
        using (myList)
        {
            myList.AutoGrow = false;

            /*
            for (int a = 0; a < (2048L * 1024L * 2048L); a++)
            {
                myList[a] = a;
            }
            */

            myList[12456] = 8;
            myList[1939848234] = 1;
            // etc...
        }