Getting "unable to complete request: one or more nodes were unavailable" when performing insert statement using cqlsh

Question

I'm trying to perform an insert on a brand new install of Cassandra 2, and while I was able to set up a new keyspace and table just fine, I get the eror mentioned above when attempting to perform an insert.

I dont' have any fancy multi server setup, it's just running one one computer with a test db hence my confusion with node configuration

Commands used to create said items are:

CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'DC1' : 3 };
USE demodb;
CREATE TABLE users (user_name varchar, state varchar, birth_year bigint, PRIMARY KEY (user_name));
INSERT INTO users (user_name, state, birth_year) VALUES ('canadiancreed', 'PA', 1976);

Answer 1

CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'DC1' : 3 };

Is most likely your culprit. It says that data in the demodb keyspace belongs in DC1 and should be replicated 3 times. If your single test node is not specified as being a member of DC1 any request to insert to this keyspace will fail. In addition, if it is a member of DC1 and the consistency level is greater than 1 all requests will fail because it will be impossible for the write to get more than one acknolegdment.

Check what your Data Center is named (nodetool status) and adjust they keyspace replication details to match. That will most likely solve your problems.

---- Edited for more Details and Better Formatting ----

This is one of the most common errors new users have with Cassandra. Basically in Cassandra there are logical units of hardware we call Datacenters. A datacenter is supposed to represent a group of geographically or in some other way distinct group of machines. You can make many of these and protect against failure in one geographic location from causing your application to go offline.

Keyspaces are a logical structure for organizing groups of information, it would be analgous to a Database in the relational world. Each Keyspace gets to specify on which and how many machines should it replicate against. If we use the NetworkTopologyStrategy the replication is specified on a per datacenter basiss. We specify these details at creation time (although they can be modified later) using the "CREATE KEYSPACE .... WITH REPLICATION ".

In your above statement you have specified that all information within the Keyspace demodb should be placed in the datacenter "DC1" and there should be 3 copies of the data in that datacenter. This basically means you have at least 3 Nodes in DC1 and you want a copy of the data on each of those nodes. This by itself will not cause an insert to fail unless the entire datacenter is unknown to the Cassandra cluster. This would be the case if you did no initial configuration of your C* cluster and are just running off the stock yaml.

Running nodetool status will show you what a current node believes about the state of the cluster. Here is the output from C* running off my local machine.

Datacenter: Cassandra
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Owns (effective)  Host ID                               Token                                    Rack
UN  127.0.0.1  93.37 KB   100.0%            50be3bec-7e30-4385-bd4a-918055a29292  4731866028208108826                      rack1

This output shows that I have a single node operating within a cluster named "Cassandra". This means any inserts to keyspaces which require replicas in other Datacenters will fail because the cluster doesn't know how to handle those requests. (If the nodes were simply down but we had seen them before we could save hints but if the other DC has never been seen we reject the request because the cluster has most likely been misconfigured.)

To fix this situation for I would modify my Keyspace using

cqlsh:demodb> ALTER KEYSPACE demodb WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'Cassandra' : 1 };

Now demoDB requires a copy of the data in 1 machine in the datacenter Cassandra. This is great beacuase as my nodetool output states, I have one node in a datacenter named Cassandra. If I try an insert now it passes.

cqlsh:demodb> INSERT INTO users (user_name, state, birth_year) VALUES ('canadiancreed', 'PA', 1976);
cqlsh:demodb> select * from users where user_name = 'canadiancreed' ;

 user_name     | birth_year | state
---------------+------------+-------
 canadiancreed |       1976 |    PA

(1 rows)

and I would change my setup schema script to have the correct datacenter name as well

CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'Cassandra' : 1 };

Answer 2

In case you end up here after a Google search, I found that you can get this error if you are using the consistency level ALL (might also be the case for QUORUM with specific replication factor numbers) and the keyspace you use is setup to be replicated on a non-existent or dead datacenter.

Updating the keyspace replication to remove reference to the non-existent datacenter solves the issue.

(and the message is entirely logical in this case: you want results from nodes that don't exist anymore).