mmap-loadable data structure library for C++ (or C)

Question

I have a some large data structure (N > 10,000) that usually only needs to be created once (at runtime), and can be reused many times afterwards, but it needs to be loaded very quickly. (It is used for user input processing on iPhoneOS.) mmap-ing a file seems to be the best choice.

Are there any data structure libraries for C++ (or C)? Something along the line

ReadOnlyHashTable<char, int> table ("filename.hash");
// mmap(...) inside the c'tor
...
int freq = table.get('a');
...
// munmap(...); inside the d'tor.

Thank you!

---

Details:

I've written a similar class for hash table myself but I find it pretty hard to maintain, so I would like to see if there's existing solutions already. The library should

• Contain a creation routine that serialize the data structure into file. This part doesn't need to be fast.
• Contain a loading routine that mmap a file into read-only (or read-write) data structure that can be usable within O(1) steps of processing.
• Use O(N) amount of disk/memory space with a small constant factor. (The device has serious memory constraint.)
• Small time overhead to accessors. (i.e. the complexity isn't modified.)

Assumptions:

• Bit representation of data (e.g. endianness, encoding of float, etc.) does not matter since it is only used locally.
• So far the possible types of data I need are integers, strings, and struct's of them. Pointers do not appear.

P.S. Can Boost.intrusive help?

Answer 1

You could try to create a memory mapped file and then create the STL map structure with a customer allocator. Your customer allocator then simply takes the beginning of the memory of the memory mapped file, and then increments its pointer according to the requested size. In the end all the allocated memory should be within the memory of the memory mapped file and should be reloadable later.

You will have to check if memory is free'd by the STL map. If it is, your customer allocator will lose some memory of the memory mapped file but if this is limited you can probably live with it.