RabbitMQ Shovel plugin stuck on "starting" status

Question

RabbitMQ starts up just fine, but the shovel plugin status is listed as "starting".

I'm using the following rabbitmq.config:

Each broker is running on a separate AWS instance. The remote server is windows 2008 server, the local server is Amazon Linux.

[{rabbitmq_shovel,
  [{shovels,
    [{scrape_request_shovel,
      [{sources,      [{broker,"amqp://test_user:test_password@localhost"}]},
       {destinations, [{broker, "amqp://test_user:test_password@ec2-###-##-###-###.compute-1.amazonaws.com"}]},
       {queue, <<"scp_request">>},
       {ack_mode, on_confirm},
       {publish_properties, [{delivery_mode, 2}]},
       {publish_fields, [{exchange, <<"">>},
                         {routing_key, <<"scp_request">>}]},
       {reconnect_delay, 5}
       ]}
      ]
   }]
}].

Running the following command:

sudo rabbitmqctl eval 'rabbit_shovel_status:status().'

returns:

[{scrape_request_shovel,starting,{{2012,7,11},{23,38,47}}}]

According to This question, this can result if the users haven't been set up correctly on the two brokers. However, I've double-checked that I've set up the users correctly via rabbitmqctl user_add on both machines -- have even tried it with a different set of users, to be sure.

I also ran an nmap scan of port 5672 on the remote host to verify is was up and running on that port.

UPDATE Problem isn't solved but this does appear to be a result of connection problems with the remote server. I changed "reconnect_delay" to 0 in my config file, to avoid having shovel infinitely re-try the connection. Highly recommend others with this problem do this as well, as it allows you to get error messages out of rabbit_shovel_status. In my case I got the following error:

[{scrape_request_shovel,
    {terminated,
        {{badmatch,{error,access_refused}},
         [{rabbit_shovel_worker,make_conn_and_chan,1},
          {rabbit_shovel_worker,handle_cast,2},
          {gen_server2,handle_msg,2},
          {proc_lib,init_p_do_apply,3}]}},
    {{2012,7,12},{0,4,37}}}]

Answer 1

Answering my own question here, in case others encounter this issue. This error (and also a timeout error if you get it, {{badmatch,{error,etimedout}}, ), is almost certainly a communications problem between the two machines, most likely due to port access / firewall settings.

There were a couple of dumb things I was doing here:

1) Was using the wrong DNS for my remote EC2 instance (D'oh! really dumb -- can't tell you how long I spent banging my head against the wall on this one...). Remember that stopping and starting your instance generates a new DNS, if you don't have an elastic IP associated with the instance.

2) My remote instance is a windows server, and I realized you have to open up port 5672 both in windows firewall and in EC2 security groups -- there are two overlapping levels of access controls here, and opening up the port in the EC2 management console isn't sufficient if your machine is windows server on EC2, as you also have to configure the windows server firewall.