Possible Duplicate:
Are they any decent on-disk implementations of Java's Map?
I have a piece of code (that I didn't write) that reads millions of CSV rows to a Map, then processes it.
I got to the point where I simply ran out of RAM
My options are
Rewrite the code, trying to stream the data, however since some calculations might need the entire data set (e.g. calculation that might need both the very first and very last row in the data set)
Write a Class that implements java.util.Map
but will persist the data into a database
Simply rewrite the code and insert / select from a database directly, but I'd rather try #2 first
So the thought of a DB backed Map all of a sudden made sense to me, so before starting to write it, I wanted to ask if there is a well known pattern / implementation for this problem (perhaps not even a Map)
Now as much as I like writing code, I don't like reinventing things, and I prefer reusing open source code.
I don't mind much about the storage implementation, SQL or NoSQL, but it needs to allow a Map to be automatically persistent, and avoid keeping it entirely in memory.
Is there such a known library / implementation? is this problem familiar? am I attacking it in the right way?
Update:
based on comments, I'll look into these (older, but pretty much duplicate) questions:
Are they any decent on-disk implementations of Java's Map?
Disk based HashMap
and vote to close this one if they answer my question and still up to date
Update2:
Many key-value stores provide Map interface. For example, https://github.com/jankotek/JDBM3
See also SO questions:
key-value store suggestion
Java disk-based key-value storage
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With