Storing many objects using Java

Question

I'm reaching out to the community here to understand if there is a way to store many objects in a Java map structure (> 500K). I know at some point my application will exceed its memory footprint if I use the Java Collections API and I'm looking for other solutions (outside of distributed caching). My intention is to store something to the effect of Map<String, List<String>>.

Requirements:

• Structure will be immutable (one time load into the map). Only read access.
• Access to the map needs to be fairly quick but not looking for low latency retrievals, more concerned about storing and retaining these objects in memory.

Does anyone know of such a library I can utilize that can achieve this or has anyone ever come across a solution in which they were required to load many objects in memory in Java? I'd be very interested in hearing your feedback.

Thanks.

Answer 1

EhCache would be perfect for this. At its most basic, it offers a key-value map, with optional overflow to disk, and optional persistence over JVM restarts. It will keep elements in memory that are most frequently used.

Answer 2

I'm with skaffman. In addition to the overflow to disk EhCache offers you to place the cache in-process but off-heap. If your cache is really big, this may have a very positive impact on performance since it reduces stress on the GC.

This specific feature however must be payed for, but otherwise EhCache is free.

Instead of EhCache, there are a couple of other caches in Java that offer similar or sometimes even more advanced options like Infinispan or OsCache.

Answer 3

Have you considered a read-only database, such as a java cdb: http://www.strangegizmo.com/products/sg-cdb/