Java collections that spool to disk

Question

I'm running out of memory in Java due to some very large Lists and Sets built up over the course of a transaction and iterated over just once at transaction's end. Are there any libraries that provide Java collections that can spool their serializable contents to disk when the collection size exceeds a given threshold?

Answer 1

You could try something like ehcache and its overflowToDisk option

Answer 2

I won't post you example code because it would become too long, but this is how I done it before:

1. Extend LinkedBlockingQueue.
2. Override its offer, put, poll, take and remove methods. Example: if superclass' offer returns false (capacity reached), then I would start serializing to disk.
3. Likewise, in take implementation, you check if you have any present elements in memory at all, and, if not, then you start reading from disk (and removing the first record read because it will now reside in memory; or you can read records in batches, of course).
4. Assign each instance of such a queue a filesystem-safe identifier, so that I can use it to create filesystem-safe file name for it. Also, to take it further, I would probably use current user's home directory as a place for these queues to be serialized onto disk.

That way, 99% of your disk-serialized queue is ready, and you only put your extra functionality at exactly the right spots. You would need to thoroughly read the documentation of Java's BlockingQueue interface, but hey, it's worth your time because you will only add that little extra bit of functionality you need, rather than writing the whole thing from scratch.

Hope this helps.