Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What will happen if write failed in cassandra cluster when using QUORUM CL?

Tags:

cassandra

Suppose i have 3 nodes, the RF is 3, and using QUORUM CL. When i write a data record to the cluster, if one node succeed, one failed. So the whole write request is failed, what will happen to the succeed node? Will it be roll back automatically? or it will be propagated to other node via gossip. And finally the 3 nodes will all have the record even the original request was failed?

like image 423
Ivan Avatar asked Jun 19 '15 09:06

Ivan


People also ask

How does Cassandra quorum work?

Quorum consistency is consistency in Cassandra for high mechanism and to ensure that how many nodes will respond when we will define the read and write consistency in Cassandra. In Quorum consistency a majority of (n/2 +1) nodes of the replicas must respond.

How is the Quorum of various replicas of data determined in Cassandra?

In a two datacenter cluster where each datacenter has a replication factor of 3, a quorum is 4 nodes. The cluster can tolerate 2 replica nodes down. In a five datacenter cluster where two datacenters have a replication factor of 3 and three datacenters have a replication factor of 2, a quorum is 7 nodes.

Can Cassandra lose data?

The data on Apache Cassandra is replicated. Although a complete failure might be rare, data might get corrupted. In some cases, the hardware might crash, and the data might be lost. Therefore, it is necessary to take regular backups by taking snapshots of all Cassandra nodes.

What does success mean for Cassandra's operation?

Success means that the data was written to the commit log and the memtable as described in how data is written. The coordinator node forwards the write to replicas of that row, and responds to the client once it receives write acknowledgments from the number of nodes specified by the consistency level.


1 Answers

shutty's answer is wrong in subtle ways though the article referred to is correct and an excellent source. The first three points appear correct:

  • Query coordinator will try to persist your write on all nodes according to RF=3. If 2 of them has failed, the CL=QUORUM write considered as failed.
  • A single node which accepted the failed write will not rollback it. It will persist it on memtable/disk as nothing suspicious happened.
  • Cassandra is an eventually consistent database, so it's absolutely fine for it to be in an inconsistent state for some period of time, but converging to consistent state in some future.

However the last two appear wrong and here's the corrected version:

  • Next time you read (CL=QUORUM) the key you previously failed to write, if there's still not enough nodes online, you'll get failed read. If the two nodes that failed to write previously are online (and not the one that succeeded the write) you'll receive previous value, unaffected by the failed write.
  • If the node that succeeded in writing also is online, a QUORUM read will result in read-repair causing the nodes that failed to write the newer value to update to the new value, then it will be returned. (Note: the word 'newer' is in the timestamp sense, so it is possible that even though the data was written more recently that it has an older timestamp -> this cluster started in an inconsistent state.)
like image 197
mbaryu Avatar answered Oct 09 '22 11:10

mbaryu