Disk based STL replacement for c++

Question

I've recently build an application that relies heavily on stl queues, and have run into memory issues, the queue grows to big to fit in the machines memory. Are there any disk-backed opensource, IO / memory efficient queue implementations that can be plugged inplace of a stl queue making my app magically use less RAM?

I would like something with read/write buffer-mechanisms - since my application does arround 50k enqueues/dequeues pr sec, and the objects are rather small.

My queue contains pairs of 32 bit integers, making every entry takeup 64 bit, linked lists are there for out of the question since they will double the memory consumption.

Answer 1

I would look into using STXXL. It re-implements many of the stl container interfaces with a filesystem backend. One of those implementations is for the deque structure.

Answer 2

Yes, please try STXXL. But don't use the deque if you can avoid random access (i.e. operator[]). Use stxxl::sequence instead, it is efficient in external memory.

Disclaimer: I am reviving the library. Please send me a patch if you fix the C++11 problem, which is probably due to MacOSX's toolchain.