Using a file as a storage medium for a large Map in java

Question

I have a large amount of data I need to store in a Map<String, int...>. I need to be able to perform the following functions:

1. containsKey(String key)
2. get(String key).add(int value)
3. put(String key, singletonList(int value))
4. entrySet().iterator()

I originally was just using a HashMap<String, ArrayList<Integer>> where singletonList is a function that creates a new ArrayList<Integer> and adds the given value to it. However, this strategy does not scale well to the amount of data I am using as it stores everything in the RAM and my RAM is not big enough to store all the data.

My next idea was to just dump everything into a file. However, this would mean that get, containsKey, and put would become very expensive operations, which is not at all desirable. Of course, I could keep everything sorted, but that is often difficult in a large file.

I was wondering if there is a better strategy out there.

Answer 1

Using an embedded database engine (key-value store), such as MapDB, is one way to go.

From MapDB's official website:

MapDB is embedded database engine. It provides java collections backed by disk or memory database store. MapDB has excellent performance comparable to java.util.HashMap and other collections, but is not limited by GC. It is also very flexible engine with many storage backend, cache algorithms, and so on. And finally MapDB is pure-java single 400K JAR and only depends on JRE 6+ or Android 2.1+.

If that works for you, you can start from here.