How to restart one live node from a multi node cassandra cluster?

Question

I have a production cassandra cluster of 6 nodes. I have made some changes to the cassandra.yaml file on one node and hence need to restart it. How can I do it without losing any data or causing any cluster related issues? Can I just kill the cassandra process on that particular node and start it again. Cluster Info: 6 nodes. All active. I am using AWS Ec2Snitch.

Thanks.

Answer 1

In case you are using replication factor greater than 1, and not using ALL consistency setting on your writes/reads, you can perform steps listed below, without any downtime/data loss. In case you have one of the limitations listed above, you'll need to increase your replication factor/change requests consistency before you continue.

1. Perform nodetool drain on that node (http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsDrain.html).
2. Stop the service.
3. Start the service.

In Cassandra, if durable writes are enabled, you should not lose data anyway - there's a mechanism of commitlog log replay in case of accidental restart, so you should not lose any data if doing just restart, but replaying commitlog can take some time.

The steps written above are a part of official upgrade procedure, and should be the "safest" option. You can do nodetool flush + restart, this will ensure that commitlog replay will be minimal and can be faster than drain approach.

Answer 2

Can I just kill the cassandra process on that particular node and start it again.

Essentially, yes. I'm assuming you have a RF of 3 with 6 nodes, so it shouldn't be a big deal. If you want, to do what I call a "clean shutdown" you could run the following commands first:

nodetool disablegossip
nodetool drain

And then (depending on your install):

sudo service cassandra stop

Or:

kill `cat cassandra.pid`

Note that if you do not complete these steps, you should still be ok. The drain just flushes the memtables to disk. If that doesn't happen, the commit log is reconciled against what's on disk at boot-time anyway. Those steps will just make your boot go faster.