How to handle an array with size 1,000,000,000 in C++?

Question

I need to handle 3D cube data. Its number of elements can be several billions. I understand I can't allocate that much memory on Windows. So I am thinking disk-based operations with in-process database. Is there any better way to do this? Maybe something in boost?

Update: I will eventually have to provide browsing functionality with plots.

Update2: The following article seemed to be a good solution using memory mapped file. I will try it and update again. http://www.codeproject.com/Articles/26275/Using-memory-mapped-files-to-conserve-physical-mem

Answer 1

The first and most basic step is to break the data down into chunks. The size of the chunk depends on your needs: it could be the smallest or largest chunk that can be drawn at once, or for which geometry can be built, or an optimal size for compression.

Once you're working with manageable chunks, the immediate memory problem is averted. Stream the chunks (load and unload/save) as needed.

During the load/save process, you may want to involve compression and/or a database of sorts. Even something simple like RLE and SQLite (single table with coordinates and data blob) can save a good bit of space. Better compression will allow you to work with larger chunk sizes.

Depending on usage, it may be possible to keep chunks compressed in-memory and only uncompress briefly on modification (or when they could be modified). If your data is read-only, loading them and uncompressing only when needed will be very helpful.

Splitting the data into chunks also has side-benefits, such as being an extremely simple form for octrees, allowing geometry generation (marching cubes and such) to run on isolated chunks of data (simplifies threading), and making the save/load process significantly simpler.