How can I use a very large dictionary in C#?

Question

I want to use a lookup map or dictionary in a C# application, but it is expected to store 1-2 GB of data.

Can someone please tell if I will still be able to use dictionary class, or if I need to use some other class?

EDIT : We have an existing application which uses oracle database to query or lookup object details. It is however too slow, since the same objects are getting repeatedly queried. I was feeling that it might be ideal to use a lookup map for this scenario, to improve the response time. However I am worried if size will make it a problem

Answer 1

Short Answer

Yes. If your machine has enough memory for the structure (and the overhead of the rest of the program and system including operating system).

Long Answer

Are you sure you want to? Without knowing more about your application, it's difficult to know what to suggest.

• Where is the data coming from? A file? Files? A database? Services?
• Is this a caching mechanism? If so, can you expire items out of the cache once they haven't been accessed for a while? This way, you don't have to hold everything in memory all the time.
• As others have suggested, if you're just trying to store lots of data, can you just use a database? That way you don't have to have all of the information in memory at once. With indexing, most databases are excellent at performing fast retrieves. You could combine this approach with a cache.
• Is the data that will be in memory read only, or will it have to be persisted back to some storage when something changes?
• Scalability - do you expect that the amount of data that will be stored in this dictionary will increase as time goes on? If so, you're going to run into a point where it's very expensive to buy machines that can handle this amount of data. You might want to look a distributed caching system if this is the case (AppFrabric comes to mind) so you can scale out horizontally (more machines) instead of vertically (one really big expensive point of failure).

UPDATE

In light of the poster's edit, it sounds like caching would go a long way here. There are many ways to do this:

• Simple dictionary caching - just cache stuff as its requested.
• Memcache
• Caching Application Block I'm not a huge fan of this implementation, but others have had success.

Answer 2

As long as you're on a 64GB machine, yes you should be able to use that large of a dictionary. However if you have THAT much data, a database may be more appropriate (cassandra is really nothing but a gigantic dictionary, and there's always MySQL).

Answer 3

When you say 1-2GB of data, I assume that you mean the items are complex objects that cumulatively contain 1-2GB.

Unless they're structs (and they shouldn't be), the dictionary doesn't care how big the items are.
As long as you have less than about 2²⁴ items (I pulled that number out of a hat), you can store as much as you can fit in memory.

However, as everyone else has suggested, you should probably use a database instead.
You may want to use an in-memory database such as SQL CE.