Storing million key-value in memcached -- good or bad idea?

Question

I am considering using memcached in conjunction with my PHP app to store 5 millions key-value pairs. My objective is to avoid back and forth from DB (which in my case is the filesystem). I may have 100-500 accesses per seconds to the key-values. The Key-values are both MD5's and are in the form:

array( 'MD5X' => 'MD5Y', ... )

I am not sure how the data is stored, but if we multiply 5 million * 16 bytes (keys) + 5 million * 16 bytes (values) we get ~180MB.

(EDIT: after trying with a real memcached instance I use up 750MB to store all items.)

The dataset is fixed so I will only read from it.

Questions:

1. Is this a good or bad design?
2. Can I force memcached to never (unless server crashes) have to reload the data? Assuming that the memory cap is higher than the data stored? If not, which techniques may I employ to achieve the same goal.

Thanks a lot!

Answer 1

Will you get the performance you need? Definitely. Memcache is blazing fast. We store about 10 million keys and we access memcache about 700 times a sec. It has never let us down.

You can load all the keys in memcache when you start the application and set the expiration date to be a very long time. The thing that you got to remember is that memcache is ultimately a cache. And it should not be used as a storage engine. You have to design it thinking that there is always a possibility of not finding the data (key) that you need, and make a DB call in that case.

The alternative you can look at a noSQL database like cassandra, It has excellent read and write speeds that should cater to your needs. The only thing is it is a bit difficult to fine tune cassandra as compared to memcache.