Cassandra UnavailableException: Cannot achieve consistency level LOCAL_ONE

Question

I have a 3-node Cassandra cluster with a keyspace that has a replication factor of 3:

CREATE KEYSPACE demo
  WITH REPLICATION = {
    'class':'SimpleStrategy',
    'replication_factor':3
  };

(Deployed in just one data center)

When doing failure testing, ie taking one node down, I'm getting these exceptions when attempting queries against my keyspace:

 Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
    at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:296) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1774) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1736) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1682) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1597) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:997) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:277) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:247) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:521) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503) ~[apache-cassandra-3.10.jar:3.10]
    ... 47 common frames omitted

I'm not sure why I'm seeing this error because:

1. My replication factor is set to 3 (ie I still have 2 nodes up that each contain all of the data)
2. My consistency level is set to QUORUM. (Why am I seeing LOCAL_ONE?)

Answer 1

The consistency level you see in the log is the consistency level Cassandra is internally using to retrieve auth information stored in system_auth keyspace.

Cassandra uses QUORUM when querying system auth for the default Cassandra user "cassandra"

Cassandra uses LOCAL_ONE when querying system_auth for other users

By default, Cassandra uses SimpleStrategy for system_auth keyspace (which is not great), and a replication factor of 1 (which is not great). It is strongly recommended to switch to NetworkTopologyStrategy instead, as well as increasing the replication factor to at least 3 per Data Center. This should resolve your issues.

Everything is explained in the Apache Cassandra documentation:

http://cassandra.apache.org/doc/latest/operating/security.html#authentication

Answer 2

So it turns out the problem in my case was with the system_auth keyspace which needed to have it's replication factor increased (to 2 or 3) in order to be highly available. This was required since I am using the PasswordAuthenticator for authentication.

I had to fix this with the following statement:

ALTER KEYSPACE "system_auth"
  WITH REPLICATION = {
    'class':'SimpleStrategy',
    'replication_factor':3
  };

(The default is 1 for replication factor)

The need to do this is documented here: https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigNativeAuth.html