Couchbase: net.spy.memcached.internal.CheckedOperationTimeoutException

Question

I'm loading local Couchbase instance with application specific json objects.

Relevant code is:

CouchbaseClient getCouchbaseClient()
{
    List<URI> uris = new LinkedList<URI>();
    uris.add(URI.create("http://localhost:8091/pools"));
    CouchbaseConnectionFactoryBuilder cfb = new CouchbaseConnectionFactoryBuilder();
    cfb.setFailureMode(FailureMode.Retry);
    cfb.setMaxReconnectDelay(1500); // to enqueue an operation
    cfb.setOpTimeout(10000); // wait up to 10 seconds for an operation to succeed
    cfb.setOpQueueMaxBlockTime(5000); // wait up to 5 seconds when trying to
                                      // enqueue an operation
    return new CouchbaseClient(cfb.buildCouchbaseConnection(uris, "my-app-bucket", ""));
}

Method to store entry (I'm using suggestions from Bulk Load and Exponential Backoff):

  void continuosSet(CouchbaseClient cache, String key, int exp, Object value, int tries)
  {
    OperationFuture<Boolean> result = null;
    OperationStatus status = null;
    int backoffexp = 0;

    do
    {
      if (backoffexp > tries)
      {
        throw new RuntimeException(MessageFormat.format("Could not perform a set after {0} tries.", tries));
      }

      result = cache.set(key, exp, value);
      try
      {
        if (result.get())
        {
          break;
        }
        else
        {
          status = result.getStatus();
          LOG.warn(MessageFormat.format("Set failed with status \"{0}\" ... retrying.", status.getMessage()));
          if (backoffexp > 0)
          {
            double backoffMillis = Math.pow(2, backoffexp);
            backoffMillis = Math.min(1000, backoffMillis); // 1 sec max
            Thread.sleep((int) backoffMillis);
            LOG.warn("Backing off, tries so far: " + tries);
          }
          backoffexp++;
        }
      }
      catch (ExecutionException e)
      {
        LOG.error("ExecutionException while doing set: " + e.getMessage());
      }
      catch (InterruptedException e)
      {
        LOG.error("InterruptedException while doing set: " + e.getMessage());
      }
    }
    while (status != null && status.getMessage() != null && status.getMessage().indexOf("Temporary failure") > -1);
  }

When continuosSet method called for a large amount of objects to store (single thread) e.g.

CouchbaseClient cache = getCouchbaseClient();
do
{
    SerializableData data = queue.poll();
    if (data != null)
    {
        final String key = data.getClass().getSimpleName() + data.getId();
        continuosSet(cache, key, 0, gson.toJson(data, data.getClass()), 100);
...

it generates CheckedOperationTimeoutException inside of continuosSet method in result.get() operation.

Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: 127.0.0.1/127.0.0.1:11210
        at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:160) ~[spymemcached-2.8.12.jar:2.8.12]
        at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) ~[spymemcached-2.8.12.jar:2.8.12]

Can someone shed light into this how to overcome and recover from this situation? Is there a good technique/workaround on how to bulk load in Java client for Couchbase? I already explored documentation Performing a Bulk Set which is unfortunately for PHP Couchbase client.

Answer 1

My suspicion is that you may be running this in a JVM spawned from the command line that doesn't have that much memory. If that's the case, you could hit longer GC pauses which could cause the timeout you're mentioning.

I think the best thing to do is to try a couple of things. First, raise the -Xmx argument to the JVM to use more memory. See if the timeout happens later or goes away. If so, then my suspicion about memory is correct.

If that doesn't work, raise the setOpTimeout() and see if that reduces the error or makes it go away.

Also, make sure you're using the latest client.

By the way, I don't think this is directly bulk loading related. It may happen owing to a lot of resource consumption during bulk loading, but it looks like the regular backoff must be working or you're not ever hitting it.