Celery gives connection reset by peer

Question

I setup the rabbitmqserver and added the users using the following steps:

uruddarraju@*******:/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.3$ sudo rabbitmqctl list_users 
Listing users ...
guest   [administrator]
phantom [administrator]
phantom1    []

sudo rabbitmqctl set_permissions -p phantom phantom1 ".*" ".*" ".*"

uruddarraju@******:/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.3$ sudo netstat -tulpn | grep :5672
tcp6       0      0 :::5672                 :::*                    LISTEN      31341/beam.smp  

My celery config is like:

BROKER_URL = 'amqp://phantom:[email protected]/phantom'

My code is like:

__author__ = 'uruddarraju'

from celery import Celery
import time
import celeryconfig

app = Celery('tasks')
app.config_from_object(celeryconfig)


@app.task
def add(x, y):
    print 'sleeping'
    time.sleep(20)
    print 'awoke'
    return x + y

When I try to run

celery -A celery worker --loglevel=info 

I get

[2014-07-08 23:30:05,028: ERROR/MainProcess] consumer: Cannot connect to amqp://phantom:**@10.98.85.92:5672/phantom:
[Errno 54] Connection reset by peer.
Trying again in 2.00 seconds...
[2014-07-08 23:30:07,101: ERROR/MainProcess] consumer: Cannot connect to amqp://phantom:**@10.98.85.92:5672/phantom:
[Errno 54] Connection reset by peer.
Trying again in 4.00 seconds...

Everything looks just perfect !! Can someone help me what I am missing here ?

Answer 1

I found the Connection rest by peer error, and assumed RabbitMQ was not listening on the port or there was a firewall blocking it. That was not the case.

Running RabbitMQ on Ubuntu 16.04.2 LTS, version installed by apt: 3.5.7-1ubuntu0.16.04.1

The output of 'netstat -a' for Rabbit's port 5672 looked like:

$ netstat -a|grep -i amqp
tcp6       0      0 [::]:amqp               [::]:*                  LISTEN

However, Rabbit was in fact listening on both the IPv4 address and the IPv6 address. I verified this with netcat.

To reach Rabbit from another machine, you have to set up a user account with something like the following commands:

sudo rabbitmqctl add_user celery celerypassword
sudo rabbitmqctl add_vhost celery_vhost
sudo rabbitmqctl set_user_tags celery celerytag
sudo rabbitmqctl set_permissions -p celery_vhost celery .* .* .*

The issue was my URL. Mine looked like:

BROKER_URL = 'amqp://celery:[email protected]//'

After that penultimate forward slash is the name of the vhost. Above, I gave my vhost the name celery_vhost. To connect, I needed to change the URL to:

BROKER_URL = 'amqp://celery:[email protected]/celery_vhost'

I discovered this looking at the Rabbit logs in /var/log/rabbitmq. There's a .log file that contained the following error when I had the wrong URL:

{handshake_error,opening,0,
{amqp_error,access_refused,
"access to vhost '/' refused for user 'celery'",
'connection.open'}}

Answer 2

The problem was solved once I deployed my code to production, and upgrading the celery to 3.1.12. I inititally used 3.0.19 but later upgraded and did not find any issues. Thanks for the support.