Maximum capacity of Collection<T> different than expected for x86

Question

The main question in about the maximum number of items that can be in a collection such as List. I was looking for answers on here but I don't understand the reasoning.

Assume we are working with a List<int> with sizeof(int) = 4 bytes... Everyone seems to be sure that for x64 you can have a maximum 268,435,456 int and for x86 a maximum of 134,217,728 int. Links:

• List size limitation in C#
• Where is the maximum capacity of a C# Collection<T> defined?
• What's the max items in a List<T>?

However, when I tested this myself I see that it's not the case for x86. Can anyone point me to where I may be wrong?

//// Test engine set to `x86` for `default processor architecture`
[TestMethod]
public void TestMemory()
{
    var x = new List<int>();

    try
    {
        for (long y = 0; y < long.MaxValue; y++)
        x.Add(0);
    }
    catch (Exception)
    {
        System.Diagnostics.Debug.WriteLine("Actual capacity (int): " + x.Count);
        System.Diagnostics.Debug.WriteLine("Size of objects: " + System.Runtime.InteropServices.Marshal.SizeOf(x.First().GetType())); //// This gives us "4"
    }
}

For x64: 268435456 (expected)

For x86: 67108864 (2 times less than expected)

Why do people say that a List containing 134217728 int is exactly 512MB of memory... when you have 134217728 * sizeof(int) * 8 = 4,294,967,296 = 4GB... what's way more than 2GB limit per process. Whereas 67108864 * sizeof(int) * 8 = 2,147,483,648 = 2GB... which makes sense.

I am using .NET 4.5 on a 64 bit machine running windows 7 8GB RAM. Running my tests in x64 and x86.

EDIT: When I set capacity directly to List<int>(134217728) I get a System.OutOfMemoryException.

EDIT2: Error in my calculations: multiplying by 8 is wrong, indeed MB =/= Mbits. I was computing Mbits. Still 67108864 ints would only be 256MB... which is way smaller than expected.

Answer 1

The underlying storage for a List<T> class is a T[] array. A hard requirement for an array is that the process must be able to allocate a contiguous chunk of memory to store the array.

That's a problem in a 32-bit process. Virtual memory is used for code and data, you allocate from the holes that are left between them. And while a 32-bit process will have 2 gigabytes of memory, you'll never get anywhere near a hole that's close to that size. The biggest hole in the address space you can get, right after you started the program, is around 500 or 600 megabytes. Give or take, it depends a lot on what DLLs get loaded into the process. Not just the CLR, the jitter and the native images of the framework assemblies but also the kind that have nothing to do with managed code. Like anti-malware and the raft of "helpful" utilities that worm themselves into every process like Dropbox and shell extensions. A poorly based one can cut a nice big hole in two small ones.

These holes will also get smaller as the program has been allocating and releasing memory for a while. A general problem called address space fragmentation. A long-running process can fail on a 90 MB allocation, even though there is lots of unused memory laying around.

You can use SysInternals' VMMap utility to get more insight. A copy of Russinovich's book Windows Internals is typically necessary as well to make sense of what you see.

Answer 2

This could maybe also help but i was able to replicate this 67108864 limit by creating a test project with the provided code

in console, winform, wpf, i was able to get the 134217728 limit

in asp.net i was getting 33554432 limit

so in one of your comment you said [TestMethod], this seem to be the issue.