Fastest, non-memory-based, multi-process key-value store for Node.js

Question

What is the fastest non-memory key-value store for Node.js supporting multiple processes?

I need to store simple key-value string/string pairs (not documents or JSON, just strings).
Here are some examples (there would be millions of those):

• 12345678 – abcdefghijklmnopabcdefghijklmnop
• 86358098 – ahijklmnopbcdefgahijklmnopbcdefg
• abcdefghijklmnopabcdefghijklmnop - 12345678
• ahijklmnopbcdefgahijklmnopbcdefg - 86358098

I have tried:

• Redis: it's really fast and does everything I need, but consumes too much RAM.
• LevelDB: it's fast and not too heavy on RAM, but only single-process.

A workaround for LevelDB is multilevel, which exposes a single LevelDB process though HTTP.
But that of course comes at a cost; I need something fast.

Is there any key-value store that:

• supports Node.js or has bindings for it;
• stores string/string pairs;
• supports multiple processes;
• does not entirely reside in memory;
• is fast?

I only care about reading. Fast multi-process reading is necessary, but not writing.
I'm happy with the current speed of LevelDB, just not with the fact that it is single-process.

---

Additional details:

• I'm talking about some 50 million key/value pairs, with keys and values between 8 and 500 chars.
• The code will run on a regular Linux server.
• Memory usage should be limited to a few gigabytes (4GB is fine, 8GB is acceptable)
• Reading will happen way more than writing; actually, I could do without writing.
• Speed is more important than anything (given memory and multi-process constraint are respected).

Answer 1

I would suggest to have a look at LMDB (which is the most efficient engine for OpenLDAP, and used in a number of other open-source projects).

LMDB is an embedded key/value store, with a Berkeley-DB or LevelDB like API, does not have to store everything in memory, and can support access from multiple processes. There are Node.js bindings:

• https://github.com/Venemo/node-lmdb
• https://github.com/rvagg/lmdb (see also this post)