new cassandra node can't gossip with seed

Question

I am trying to spin up a new node using cassandra 2.0.7. Both nodes are at Digital Ocean. The seed node is up and running and I can telnet to port 7000 on that host from the node I'm trying to start.

[root@cassandra02 apache-cassandra-2.0.7]# telnet 10.10.1.94 7000

Trying 10.10.1.94...

Connected to 10.10.1.94.

Escape character is '^]'.

But when I start cassandra on the new node I see the following exception:

INFO 00:01:34,744 Handshaking version with /10.10.1.94

ERROR 00:02:05,733 Exception encountered during startup

java.lang.RuntimeException: Unable to gossip with any seeds

    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1193)

    at         org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:447)

    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:656)

    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)

    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:505)

    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362)

    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:480)

    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:569)

java.lang.RuntimeException: Unable to gossip with any seeds

    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1193)

    at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:447)

    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:656)

    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)

    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:505)

    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362)

    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:480)

    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:569)

Exception encountered during startup: Unable to gossip with any seeds

ERROR 00:02:05,742 Exception in thread Thread[StorageServiceShutdownHook,5,main]

java.lang.NullPointerException

    at org.apache.cassandra.gms.Gossiper.stop(Gossiper.java:1270)

    at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:573)

    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

    at java.lang.Thread.run(Thread.java:745)

I'm using the murmur3 partition on both nodes and I have the seed node's IP listed in the cassandra.yaml of the new node. I'm just wondering what the issue might be and how I can get around it.

Answer 1

Ok, after spending a whole day at this, I eventually found the #cassandra IRC channel on freenode. (that's my first advice, go there, regularly)

The issue that you're experiencing is most likely (and was, in my case) what's called a 'chicken and the egg' problem. Chicken and the egg: Node 1: seeds=node2 Node 2: seeds=node1

Neither node can boot properly, because there's no seed node that's fully booted at.. boot time.

To solve this, just set: Node 1: seeds = node1,node2 Node 2: seeds = node2.

Now, node1 will boot. Node 2 will eventually boot as well..

Voila.

Answer 2

Check the Fire wall : Port setting in the new node

http://www.datastax.com/documentation/cassandra/2.0/cassandra/security/secureFireWall_r.html

Answer 3

The real issue here is that you're attempting to spin up Cassandra with no data and auto_bootstrap enabled on all nodes.

To JRun's point "chicken and egg" is really a "I'm trying to auto bootstrap nodes with no nodes to tell it what to do" situation.

You need to disable auto_bootstrap in your cassandra.yaml file on one of the nodes (preferably a seed).

Please see: DataStax Cassandra 2.1 Documentation

Specifically: auto_bootstrap: false (Add this setting only when initializing a fresh cluster with no data.)

So, to resolve this... Simply modify your cassandra.yaml file similar to the below example:

Three Node C* Cluster WITH NO DATA

IP Address Info:

• Node 1: 192.168.1.10
• Node 2: 192.168.1.11
• Node 3: 192.168.1.12

Originally Designated Seeds:

Node 1 (192.168.1.10) / Node 3 (192.168.1.12)

cassandra.yaml -> - seeds: "192.168.1.10,192.168.1.12"

auto_bootstrap is enabled by default

Newly Designated Seed: Node 3 (192.168.1.12)

**Make this change on all three nodes: **

cassandra.yaml -> - seeds: "192.168.1.12"

Make this change on the seed node, in addition to the above cassandra.yaml -> auto_bootstrap: false

(More than likely, the auto_bootstrap entry won't exist, so go ahead and throw it in the yaml somewhere).

After making the above changes, restart all of the cassandra processes.

You should be fine, once they all fully start up and you'll be able to go back and adjust your seeds the way you'd like!