Why Would an Out of Memory Exception be Thrown if Memory is Available?

Question

I have a fairly simple C# application that has builds a large hashtable. The keys of this hashtable are strings, and the values are ints.

The program runs fine until around 10.3 million items are added to the hashtable, when an out of memory error is thrown on the line that adds an item to the hasbtable.

According to the task manager, my program is only using 797mb of memory, and there's still over 2gb available. It's a 32-bit machine, so I know only a total of 2gb can be used by one process, but that still leaves about 1.2gb that the hashtable should be able to expand into.

Why would an out of memory error be thrown?

Answer 1

In theory you get 2GB for the process, but the reality is that it's 2GB of contiguous memory, so if your process' memory is fragmented you get less than that.

Additionally I suspect the hash table like most data structures by default will double in size when it needs to grow thus causing a huge growth when the tipping point item is added.

If you know the size that it needs to be ahead of time (or have a reasonable over-estimate) it may help to specify the capacity in the constructor.

Alternatively if it's not crucial that it's in memory some sort of database solution may be better and give you more flexibility if it does reach the point that it can't fit in memory.