Retry connection to Cassandra node upon startup

Question

I want to use Docker to start my application and Cassandra database, and I would like to use Docker Compose for that. Unfortunately, Cassandra starts much slower than my application, and since my application eagerly initializes the Cluster object, I get the following exception:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/172.18.0.2:9042 (com.datastax.driver.core.exceptions.TransportException: [cassandra/172.18.0.2:9042] Cannot connect))
    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:233)
    at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
    at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1454)
    at com.datastax.driver.core.Cluster.init(Cluster.java:163)
    at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:334)
    at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:309)
    at com.datastax.driver.core.Cluster.connect(Cluster.java:251)

According to the stacktrace and a little debugging, it seems that Cassandra Java driver does not apply retry policies to the initial startup. This seems kinda weird to me. Is there a way for me to configure the driver so it will continue its attempts to connect to the server until it succeeds?

Answer 1

You should be able to write some try/catch logic on the NoHostAvailableException to retry the connection after a 5-10 second wait. I would recommend only doing this a few times before throwing the exception after a certain time period where you know that it should have started by that point.

Example pseudocode

Connection makeCassandraConnection(int retryCount) {
    Exception lastException = new IllegalStateException();
    while (retryCount > 0) {
        try {
            return doConnectionStuff();
        } catch (NoHostAvailableException e) {
            lastException = e;
            retryCount--;
            Thread.sleep(TimeUnit.SECONDS.toMillis(5));
        }
    }
    throw lastException;
}

Answer 2

If you don't want to change your client code, and your client application's docker container stops because of the error you can use the following attribute for the client app in your docker-compose file.

restart: unless-stopped

That will restart your client application container as many times as it fails. Example docker-compose.yml file:

version: '2'
services:
  cassandra:
    image: cassandra:3.5
    ports:
      - "9042:9042"
      - "9160:9160"
    environment:
      CASSANDRA_CLUSTER_NAME: demo
  app:
    image: your-app
    restart: unless-stopped

Answer 3

The Datastax driver cannot be configured this way.

If this is only a problem with Docker and you do not wish to change your code, you could consider using something such as wait-for-it which is a simple script which will wait for a TCP port to be listening before starting your application. 9042 is cassandra's native transport port.

Other options are discussed here in the docker documentation, but I personally have only used wait-for-it but found it to be useful when working with cassandra within docker.