Cannot achieve consistency level ONE: info={ 'required_replicas': 1, 'alive_replicas': 0, 'consistency': 1}

Question

It looks like keyspace replication is not happening correctly in my Cassandra setup, I need some ideas in troubleshooting this. I have configured multi datacenter cluster but to begin with I have set the keyspace to use SimpleStrategy with RF 3.

The column families exist:

cqlsh:kairosdb> select columnfamily_name from system.schema_columnfamilies where keyspace_name = 'kairosdb';

 columnfamily_name
-------------------
       data_points
     row_key_index
      string_index

(3 rows)

but I am unable to query on them:

cqlsh:kairosdb> select count(*) from data_points limit 100000;
 Traceback (most recent call last):
  File "/usr/bin/cqlsh", line 957, in perform_simple_statement
    rows = self.session.execute(statement, trace=self.tracing_enabled)
  File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.1.post.zip/cassandra-driver-2.1.1.post/cassandra/cluster.py", line 1282, in execute
    result = future.result(timeout)
  File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.1.post.zip/cassandra-driver-2.1.1.post/cassandra/cluster.py", line 2776, in result
    raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 1}

This is how I setup the multi datacenter setup:

1. Use 2 nodes from each DC in the seeds list
2. Use org.apache.cassandra.locator.GossipingPropertyFileSnitch for the Snitch
3. Specify different name for DC in the cassandra-rackdc.properties as appropriate for the node

Here's the keyspace is created:

cqlsh:kairosdb> describe keyspace kairosdb;

CREATE KEYSPACE kairosdb WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

Any ideas what I can do to troubleshoot this?

Answer 1

In your keyspace creation you have this syntax:

CREATE KEYSPACE kairosdb WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

However if you wish to use replication across data centers (DC) then you need to use NetworkTopologyStrategy, so for example:

CREATE KEYSPACE kairosdb WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': 3, 'DC2': 3};

As per the following documentation link, "Use NetworkTopologyStrategy when you have (or plan to have) your cluster deployed across multiple data centers..."

http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html