Using an Increment counter for unique key generation in a Couchbase cluster

Question

The most common question I hear about Couchbase and other NoSQL databases is how to generate unique keys for records - or more specificly - how to replicate the AUTO INCREMENT feature of common Relation Databases.

The solution in Couchbase often mentioned is the increment function, where you can call increment on a numeric key and it will generate a new unique number in sequence.

My question on this is I cannot get my head around the massive problem I foresee when it comes to replication.

Consider you have a cluster of three Couchbase nodes and you are storing a request log. You want to key this log so you create an entry called "requestlog_counter".

Now say we we have 4 web nodes each receiving 20 requests a second and each of those needs to be recorded as "request::{ID})". That's 80 requests a second.

Say nodes 1 and 3 are having a tiny bit of network latency but both receive one of these 40 requests at precisely the same time. Your script increments the request counter (lets say for this example it is currently at 1500) and gets an ID. Surely it is now possible that BOTH Couchbase instances could return 1501 to web nodes 1 and 3 and both servers will now attempt to store the request they are dealing with as "request:1501".

Now, replication will deal with this and essentially the latest one will win. But you now have lost the record of one request.

So does that now mean that in reality you need a better way of keying vital data and that using auto increments for absolute values and unique key generation is something that should be avoided in a NoSQL cluster environment?

Or - is there something you can do as part of your key generating procedure that makes this 100% reliable.

Please also consider a multiple cluster environment using cross data centre replication too.

Thanks.

Mike

Answer 1

First of all according to couchbase documentation functions increment and decremant are "atomic" within cluster. So if you use them to generate "autoincrement" all should work fine.

But if you want to enshure that while you save new item in the couchbase you will not override existing (situation like "BOTH Couchbase instances could return 1501") you can use store operation with StoreMode.Add. So if you call at the same time couchbase.store(StoreMode.Add, "request:1501",value), one request will finish with success, other will fail and you can catch this "fail" and try to repeat that store operation again (with getting new autoincremented id for new key)