Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Create my own text based data storage

I need to store maybe 250 million records of String -> String mappings and a database seems overload for this sort of thing.

I can't store this in memory as it will run out of heap space. I could extend the heap space, but I'd prefer not to do this.

I want as fast access as I can get to a single mapping as possible and wanted to know the best way to go about this.

The initial String is unique and what I will be searching by. I also know the range the string length will lie in, but its mapped value could be any length.

Is it possible to get something as fast as a database for searching? I'm guessing not but just wanted to be sure.

I'm working in Java, but I assume this will be language independent.

like image 927
Cheetah Avatar asked Oct 08 '22 07:10

Cheetah


2 Answers

Redis http://redis.io/ is what you need!

like image 144
Ali Avatar answered Oct 13 '22 09:10

Ali


Can't you use an already existing key-value store like Voldemort, CouchDB or Cassandra? There are so many existing projects, why write a new one?

like image 42
Marek Sapota Avatar answered Oct 13 '22 11:10

Marek Sapota