Rabbitmq cluster does not work on ec2

Question

Clustering with rabbit does not work. I mean wow..I have all ports open. I am on ubuntu 12.04 and yet I get the below? Any rabbit dev ever read these posts...why does this happen? Why do the docs reflect how to cluster properly? Both have the same cookies.

thanks

sudo rabbitmqctl stop_app
rabbitmqctl join_cluster --ram [email protected]





Clustering node 'rabbit@ip-172-31-2-103' with '[email protected]' ...
Error: unable to connect to nodes ['[email protected]']: nodedown

=ERROR REPORT==== 26-Aug-2014::07:25:21 ===
** System NOT running to use fully qualified hostnames **
** Hostname ip-172-31-12-135.us-west-1.compute.internal is illegal **

DIAGNOSTICS
===========

attempted to contact: ['[email protected]']

[email protected]:
  * connected to epmd (port 4369) on ip-172-31-12-135.us-west-1.compute.internal
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?

current node details:
- node name: 'rabbitmqctl20516@ip-172-31-2-103'
- home dir: /var/lib/rabbitmq
- cookie hash: deaU3MfVotDW9r05xrIWwA==

Answer 1

Answered on the mailing list: https://groups.google.com/d/msg/rabbitmq-users/9P-BAwGVHJU/fwOpZPJywwYJ, including my response here.

** System NOT running to use fully qualified hostnames ** ** Hostname ip-172-31-12-135.us-west-1.compute.internal is illegal **

There are 3 most common issues:

• Host names: see "Issues with hostname" on http://www.rabbitmq.com/ec2.html
• Firewalls, port access: see "Firewalled nodes" on http://www.rabbitmq.com/clustering.html
• Different Erlang versions across the cluster: "If using clustered nodes, all nodes should use the same version of Erlang" on http://www.rabbitmq.com/which-erlang.html

so I'm not sure it's fair to claim that the docs are unhelpful.

Your issue seems to be 1 or 2, although all 3 need to be checked to be sure. We'll try to cross link the pages above better.

Also, a quick search for the error message above yields multiple results, e.g.:

• http://markmail.org/thread/2tgytqbittfvb2jq
• http://markmail.org/thread/qfpphcemg73luf4j
• http://markmail.org/thread/2f5alpmgwn2xybvj

which may clarify some of the issues in a bit more detail.

Answer 2

Sorry to revive this but I ran into something similar on Windows with Rabbit 3.4.2 and Erlang OTP 17.3. I'm certain this is also an issue with Rabbit back to at least 3.3.5.

My goal was to setup a RabbitMq cluster on the same vnet. The machines can see each other and can get to each others shares, etc. Machines had the same erlang cookies and reported no errors. I could connect to each broker but not get them to cluster together.

There's not a lot of help on this around the web so after struggling with it for (many) hours here's how I fixed it. For me it was a casing issue. My vms were named rabbitMq00 and rabbitMq99 so my cluster command from rabbitMq00 was:

rabbitmqctl join_cluster rabbit@rabbitMq99

Wrong! Error message that produced was just as in the original question:

rabbit@rabbitMq99:
  * connected to epmd (port 4369) on rabbitMq99
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?

Windows/Erlang/WhoKnows wants caps (probably because of the NetBIOS of olden days). The proper command is:

rabbitmqctl join_cluster rabbit@RABBITMQ99

Server name must be uppercase. Sad but true. Hopefully this can help someone.