Cassandra Nodes DN - Simple 2 Node Cluster Configuration Fail

Question

I created a 2 node data-center, cluster. This took quite a bit of work because of some confusing and out of date documentation. My problem is now, when I use the nodetool status command the remote machine's state is DN or Down Normal. From my understanding the state should be UN or Up Normal.

Also, what is interesting, what is interesting is that I receive this constant server output:

WARN  17:31:51 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:31:51 Setup task failed with error, rescheduling
WARN  17:32:01 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:01 Setup task failed with error, rescheduling
WARN  17:32:11 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:11 Setup task failed with error, rescheduling
WARN  17:32:21 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:21 Setup task failed with error, rescheduling
WARN  17:32:31 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:31 Setup task failed with error, rescheduling
WARN  17:32:41 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:41 Setup task failed with error, rescheduling
WARN  17:32:51 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:51 Setup task failed with error, rescheduling
WARN  17:33:01 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:01 Setup task failed with error, rescheduling
WARN  17:33:11 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:11 Setup task failed with error, rescheduling
WARN  17:33:21 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:21 Setup task failed with error, rescheduling
WARN  17:33:31 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:31 Setup task failed with error, rescheduling
WARN  17:33:41 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:41 Setup task failed with error, rescheduling
WARN  17:33:51 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:51 Setup task failed with error, rescheduling
WARN  17:34:01 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:34:01 Setup task failed with error, rescheduling

This output is on the machine that is not the seed. The main seed does not output this constantly.

Also, if it was not clear. If I am running the nodetool status program from the machine. The machine I am running it on is Up Normal but the remote machine is always Down Normal. But both machines show that their status is Up Normal. The machine I run the command from lists its address as the loopback address and the remote address as the 10.x.x.x type of internal enterprise LAN address. The remote machine always shows DN.

Please see the below screenshot:

Please let me know if I need to provide more information.

Thank-you for reading this.

Respectfully,

me

Answer 1

To others who may have googled this and landed here:

I ran into this issue while trying to setup a 2 node version 3.9 cluster. Each one of my hosts had two interfaces, each host connected to a gigabit switch (which is connected to other servers) and the hosts additionally connected directly to each other via a 10-gig SFP+ cable. The idea being that other servers could query the cluster via the gigabit switch but the clusters themselves could communicate via 10-gig.

After trying many different combinations of listen_address and broadcast_address and broadcast_rpc_address I managed to solve this by setting the following (based on a default cassandra.yaml):

listen_address: (private 10-gig IP)
#broadcast_address: keep this line commented out
rpc_address: 0.0.0.0
broadcast_rpc_address: (gigabit switch IP)

Running a nodetool status should show the private IPs and show they are both up. Connecting to one of the hosts via the Java driver (3.1.3) and calling getAllHosts will show both hosts as being available on their gigabit switch IP.