What's the most efficient way to manage large amounts of data (height data) and replace this huge array?

Question

I need to be able to look up this data quickly and need access to all of this data. Unfortunately, I also need to conserve memory (several of this will cause OutofMemoryExceptions)

short[,,] data = new short[8000,8000,2];

I have attempted the following:

• tried jagged array - same memory problems
• tried breaking into smaller arrays - still get memory issues
• only resolution is to map this data efficiently using a memory mapped file or is there some other way to do this?

Answer 1

How about a database? After all, they are made for this.

I'd suggest you take a look at some NoSQL database. Depending on your needs, there are also in-memory databases [which obviously could suffer from the same out-of-memory problem] and databases that can be copy deployed or linked to your application.

I wouldn't want to mess with the storage details manually, and memory-mapping files is what some databases (at least MongoDB) are doing internally. So essentially, you'd be rolling your own DB, and writing a database is not trivial -- even if you narrow down the use case.

Redis or Membase sound like suitable alternatives for your problem. As far as I can see, both are able to manage the RAM utilization for you, that is, read data from the disk as needed and cache data in RAM for fast access. Of course, your access patterns will play a role here.

Keep in mind that a lot of effort went into building these DBs. According to Wikipedia, Zynga is using Membase and Redis is sponsored by VMWare.