Couchbase cluster - exception when 1 node is down

Question

I have a Couchbase cluster with 3 nodes. I have a java process that writes to this cluster. When one of the nodes is down I'm getting this exception:

2013-12-30 14:03:23.259 WARN com.couchbase.client.CouchbaseConnection:  Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 980d330b2a96656a93bd08e48e6fc759135e6e6f.
Dec 30, 2013 2:03:23 PM com.couchbase.client.CouchbaseConnectionFactory resubConfigUpdate
INFO: Attempting to resubscribe for cluster config updates.
Dec 30, 2013 2:03:23 PM com.couchbase.client.CouchbaseConnectionFactory$Resubscriber run
INFO: Reconnect attempt 1, waiting 0ms
2013-12-30 14:03:24.020 INFO com.couchbase.client.CouchbaseConnection:  Reconnecting {QA sa=10.223.224.79/10.223.224.79:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2013-12-30 14:03:24.261 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl:  Operation canceled because authentication or reconnection and authentication has taken more than one second to complete.
2013-12-30 14:03:24.263 WARN com.couchbase.client.CouchbaseConnection:  Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 980d330b2a96656a93bd08e48e6fc759135e6e6f.
2013-12-30 14:03:25.265 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl:  Operation canceled because authentication or reconnection and authentication has taken more than one second to complete.
441016302 [pool-5-thread-1] ERROR com.polimo.rtb.writer.CouchbaseWriter  - Error while trying to write to couchbase
java.util.concurrent.CancellationException: Cancelled
        at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:176)
        at net.spy.memcached.MemcachedClient.mutateWithDefault(MemcachedClient.java:1842)
        at net.spy.memcached.MemcachedClient.incr(MemcachedClient.java:1788)
        at com.polimo.rtb.writer.CouchbaseWriter.incrementCounterInDb(CouchbaseWriter.java:107)
        at com.polimo.rtb.writer.CouchbaseWriter.write(CouchbaseWriter.java:90)
        at com.polimo.rtb.consumer.ConsumerThread.run(ConsumerThread.java:40)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

I'm using a standard java client and with the described behavior I feel I'm missing the whole point of using a cluster, I mean, if one node is down everything should be working still.

Is there some configuration I'm missing (server or client)?

Answer 1

Replication is configured on a per-bucket basis. If no replicas are configured then when a node goes down the vBucket(s) hosted by that node will be no longer be available, and hence any attempt to perform operations on that data fail as you saw.

See the Creating and Editing Data Buckets in the Couchbase Manual for details on configuring replicas (specifically the Replicas section of the "Create Bucket" process and the Failover Nodes section).