How to handle recovery memcached nodes when using spymemcached & HashAlgorithm.KETAMA_HASH

Question

I am using spymemcached & HashAlgorithm.KETAMA_HASH to connect to a pool of memcached of 5 nodes.

My understanding is when we use a consistent hashing algorithm like, when a node is down, we don't need to worry as the key will be re-distributed (with min. impact)

What if when the down-ed node is going to join the pool. What I need to do?

Should I make sure stale data need to be removed? Or should my program need special handling for this case?

Answer 1

Given that this document is accurate: http://info.couchbase.com/rs/northscale/images/Couchbase_WP_Dealing_with_Memcached_Challenges.pdf

If there is any network disruption, and one or more clients decide that a particular memcached server is not available anymore, they will automatically rehash some data into the rest of the nodes even if the original one is still available. If the node eventually returns to service (for example after the network outage is resolved), the data on that node will be out of date and the clients without updated key-server remapping info will read stale data.

Assuming this is still up to date: http://lists.danga.com/pipermail/memcached/2007-April/003852.html It would be safe refresh/flush the node before adding it back. Forcing the down-ed node to clear up any stale entry.