how to rebalance cassandra cluster after adding new node

Question

I had a 3 node cassandra cluster with replication factor of 2. The nodes were running either dsc1.2.3 or dsc1.2.4. Each node had num_token value of 256 and initial_token was commented. This 3 node cluster was perfectly balanced i.e. each owned around 30% of the data.

One of the nodes crashed so I started a new node and nodetool removed the node that had crashed. The new node got added to the cluster but the two older nodes have most of the data now (47.0% and 52.3%) and the new node has just 0.7% of the data.

The output of nodetool status is

Datacenter: xx-xxxx
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns   Host ID                               Rack
UN  10.xxx.xxx.xxx    649.78 MB  256     47.0%  ba3534b3-3d9f-4db7-844d-39a8f98618f1  1c
UN  10.xxx.xxx.xxx    643.11 MB  256     52.3%  562f7c3f-986a-4ba6-bfda-22a10e384960  1a
UN  10.xxx.xxx.xxx    6.84 MB    256     0.7%   5ba6aff7-79d2-4d62-b5b0-c5c67f1e1791  1c

How do i balance this cluster?

Answer 1

You didn't mention running a repair on the new node, if indeed you haven't yet done that it's likely the cause of your lack of data on the new node.

Until you run a nodetool repair the new node will only hold the new data that gets written to it or the data that read-repair pulls in. With vnodes you generally shouldn't need to re-balance, if I'm understanding vnodes correctly, but I haven't personally yet moved to using vnodes so I may be wrong about that.

Answer 2

It looks like your new node hasn't bootstrapped. Did you add auto_bootstrap=true to your cassandra.yaml?

If you don't want to bootstrap, you can run nodetool repair on the new node and then nodetool cleanup on the two others until the distribution is fair.