#Cassandra - What is the difference between nodetool removenode , decommission, assassinate, replace?

Question

In order to remove a node (working / non working) which command is applicable in which situation?

Answer 1

Decommission streams data from the leaving node. Therefore, you are guaranteed to maintain the same consistency that you had at the time you start the operation.

Removenode streams data from any available node that owns the range. It's possible you could violate consistency (and even potentially lose data) depending on your time since repair and the consistency level you use to write data.

Typically, you should prefer decommission if possible. Imagine you have 5 node cluster (A-E), and a given write would go to B, C, and D. You write with quorum, so for some reason C is down, but a write goes to B and D. When C comes back online, D needs to be removed from the cluster (you're downsizing, or changing drives, or something).

• If D is online, you run decommission, and D sends its data to E - you keep 2 replicas of all of the data, and you'll be able to run repair later to get the write to C.

• If D is offline, or if you run removenode instead of decommission, you may stream from C instead of D. In this case, you now have 2 replicas (C and E) that are missing the data. If you read at quorum, you may now hit C and E instead of B, and get a result that is missing the data. A repair may solve this, as long as your original write went to more than one node, but if your original write only went to one node, you may actually lose data.

Assassinate is a rarely used tool that should only be used tool to force a node out of a cluster. No streaming is performed. The chance of data loss is significantly higher, especially if you use RF < 3 and write with CL < ALL.

Answer 2

The difference lies in how the data is moved. In decommission, the leaving node moves the data before leaving the cluster where as in remove node, the data is moved using the replicas with the same tokens in the cluster.

Decommission therefore cannot be done using a dead node.