Couchbase concurrent timeout exception : Java SDK

Question

I am working with java in a maven project. I was using couchbase 2.3.1 but in trying to resolve this issue I rolled back to 2.2.8 to no avail.

The issue I get is that while I do get date through to my couchbase cluster I am seeing alot of this:

java.lang.RuntimeException: java.util.concurrent.TimeoutException
at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:75)
at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:359)
at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:354)

Below are the settings for my couchbase environment:

CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='null', sslKeystorePassword='null', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=24, computationPoolSize=24, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, searchServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.2.8 (git: 2.2.8, core: 1.2.9), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, socketConnectTimeout=1000, dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2, dcpConnectionName=dcp/core-io, callbacksOnIoPool=false, queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false}

Im not really too sure what to look at here. As far as I can tell there should be a decent enough connection between the server where the app is running and the couchbase cluster. Any help or direction on this would be helpful. Here is a snippet from where the error is being thrown.

LockableItem<InnerVertex> lv = this.getInnerVertex(id);
lv.lock();
    try {
        String content;
        try {
            content = mapper.writeValueAsString(lv.item);
        } catch (JsonProcessingException e) {
            LOG.warning(e.getMessage());
            return;
        }
        RawJsonDocument d = RawJsonDocument.create(VertexId.toKey(id), content);
        bucket.upsert(d);
    } finally {
        lv.unlock();
    }

Answer 1

I was searching for the answer. I got there are many solutions all are talking about exception. I also checked the jar code, it is called that it is timeout exception.

Root Cause Analysis

The error occured from the following section of couchbase: https://github.com/couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/java/util/Blocking.java#L71

public static <T> T blockForSingle(final Observable<? extends T> observable, final long timeout,
    final TimeUnit tu) {
    final CountDownLatch latch = new CountDownLatch(1);
    TrackingSubscriber<T> subscriber = new TrackingSubscriber<T>(latch);

    observable.subscribe(subscriber);

    try {
        if (!latch.await(timeout, tu)) { // From here, this error occurs.
            throw new RuntimeException(new TimeoutException());
        }
} 

If the timeout kicks in, a TimeoutException nested in a RuntimeException is thrown to be fully compatible with the Observable.timeout(long, TimeUnit) behavior.

Resource Link:

http://docs.couchbase.com/sdk-api/couchbase-java-client-2.2.0/com/couchbase/client/java/util/Blocking.html

Your configuration analysis and solution:

Your couchbase environment connectionTimeout is 5000ms or 5sec, which is the default value of connection timeout.

You need to increase this value to 10000ms or greater. Your problem will be solved.

//this tunes the SDK (to customize connection timeout)
CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder()
        .connectTimeout(10000) //10000ms = 10s, default is 5s
        .build();

A full Solution

Simonbasle has given a full solution in this tutorial:

From the short log, it looks like the SDK is able to connect to the node, but takes a little much time to open the bucket. How good is the network link between the two machines? Is this a VM/cloud machine?

What you can try to do is increase the connect timeout:

public class NoSQLTest {
public static void main(String[] args) {
        try {
            //this tunes the SDK (to customize connection timeout)
            CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder()
                    .connectTimeout(10000) //10000ms = 10s, default is 5s
                    .build();
            System.out.println("Create connection");
            //use the env during cluster creation to apply
            Cluster cluster = CouchbaseCluster.create(env, "10.115.224.94");
            System.out.println("Try to openBucket");
            Bucket bucket = cluster.openBucket("beer-sample"); //you can also force a greater timeout here (cluster.openBucket("beer-sample", 10, TimeUnit.SECONDS))
            System.out.println("disconnect");
            cluster.disconnect();
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }
}

As a side note, you should always reuse the CouchbaseEnvironment, CouchbaseCluster and Bucket instances once created (usually by making them public static somewhere, or a Spring singleton, etc...). These are thread safe and should be shared (and they are expensive to create anyway).

Resource Link:

1. Couchbase connection timeout with Java SDK