ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile

Question

I'm getting this error when I try to perform celery -A draft1 beat in my terminal.

Traceback (most recent call last):
  File "/home/james/postr/env/lib/python3.5/site-packages/celery/apps/beat.py", line 107, in start_scheduler
    service.start()
  File "/home/james/postr/env/lib/python3.5/site-packages/celery/beat.py", line 558, in start
    interval = self.scheduler.tick()
  File "/home/james/postr/env/lib/python3.5/site-packages/celery/beat.py", line 279, in tick
    self.apply_entry(entry, producer=self.producer)
  File "/home/james/postr/env/lib/python3.5/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/home/james/postr/env/lib/python3.5/site-packages/celery/beat.py", line 411, in producer
    return self.Producer(self._ensure_connected(), auto_declare=False)
  File "/home/james/postr/env/lib/python3.5/site-packages/celery/beat.py", line 395, in _ensure_connected
    _error_handler, self.app.conf.broker_connection_max_retries
  File "/home/james/postr/env/lib/python3.5/site-packages/kombu/connection.py", line 405, in ensure_connection
    callback)
  File "/home/james/postr/env/lib/python3.5/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
    return fun(*args, **kwargs)
  File "/home/james/postr/env/lib/python3.5/site-packages/kombu/connection.py", line 261, in connect
    return self.connection
  File "/home/james/postr/env/lib/python3.5/site-packages/kombu/connection.py", line 802, in connection
    self._connection = self._establish_connection()
  File "/home/james/postr/env/lib/python3.5/site-packages/kombu/connection.py", line 757, in _establish_connection
    conn = self.transport.establish_connection()
  File "/home/james/postr/env/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 288, in connect
    self.drain_events(timeout=self.connect_timeout)
  File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 471, in drain_events
    while not self.blocking_read(timeout):
  File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 477, in blocking_read
    return self.on_inbound_frame(frame)
  File "/home/james/postr/env/lib/python3.5/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 481, in on_inbound_method
    method_sig, payload, content,
  File "/home/james/postr/env/lib/python3.5/site-packages/amqp/abstract_channel.py", line 128, in dispatch_method
    listener(*args)
  File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 603, in _on_close
    (class_id, method_id), ConnectionError)
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.

I'm running celery on my remote Ubuntu django server.

Any idea what the problem is?

Here's my code:

settings

CELERYBEAT_SCHEDULE = {
    'post_jobs': {
        'task': 'post.tasks.post_jobs',  # the same goes in the task name
        'schedule': crontab(minute=40),
    },
    'test_post': {
        'task': 'post.tasks.test_post',
        'schedule': crontab(minute=40),
    }
}

draft1/celery.py

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'draft1.settings')

app = Celery("draft1", broker=CELERY_BROKER_URL)
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

post/celery.py

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'draft1.settings')

app = Celery(broker=CELERY_BROKER_URL)
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

post/tasks.py

@app.task
def test_post():
    from .models import Post
    for i in Post.objects.all():
        if i.entered_category == "test":
            i.entered_category = "not_test"
            i.save()
    return HttpResponseRedirect('/')

postr-celery.conf

[program:postr-celery]
command=/home/james/postr/env/bin/celery -A post worker --loglevel=INFO
directory=/home/james/postr
user=james
numprocs=1
stdout_logfile=/var/log/supervisor/celery.log
stderr_logfile=/var/log/supervisor/celery.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

stopasgroup=true

; Set Celery priority higher than default (999)
; so, if rabbitmq is supervised, it will start first.
priority=1000

I'm not sure what sparked this error, my celery was working recently. Any idea what the problem is?

Answer 1

There is a good chance that a software update of the SSL features caused the problem. I found my issue in the /var/log/rabbitmq/[email protected] file.

In my case, I found the following, and to resolve I had to re-install my rabbitmq / celery.

/lib/erlang/lib/crypto-4.2/priv/lib/crypto: 'libcrypto.so.1.0.0: cannot open shared object file: No such file

a test of rabbitmqctl list_users and add_users can help determine if your rabbitmqctl is installed correctly when you monitor the logs.