Docker / Celery: Can't get celery to run

Question

I can't get my celery workers to constantly listen to the default queue. Celery constantly exits.

$: docker-compose up
Starting tasker_rabbitmq_1
Starting tasker_celery_1
Attaching to tasker_rabbitmq_1, tasker_celery_1
tasker_celery_1 exited with code 1
rabbitmq_1  | 
rabbitmq_1  |               RabbitMQ 3.6.1. Copyright (C) 2007-2016 Pivotal Software, Inc.
rabbitmq_1  |   ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
rabbitmq_1  |   ##  ##
rabbitmq_1  |   ##########  Logs: /var/log/rabbitmq/[email protected]
rabbitmq_1  |   ######  ##        /var/log/rabbitmq/[email protected]
rabbitmq_1  |   ##########
rabbitmq_1  |               Starting broker... completed with 6 plugins.

I am trying to build an app that has a separate jobs layer as a separately deployed jobs container. So the architecture is:

1. Web / App layer (Django) on EBS
2. Jobs Layer: Celery + RabbitMQ as a Docker container

This is what I have:

Folder structure:

-tasker
    -tasker
        -tasks.py
        -celeryconfig.py
        - __init__.py
    -Dockerfile
    -docker-compose.yml
    -requirements.txt

tasks.py:

from celery import Celery
from celery import task

celery = Celery('tasks', broker='amqp://guest@localhost//')

import os


@celery.task
def add(x, y):
    return x + y

Dockerfile:

FROM python:3.4
ENV PYTHONBUFFERED 1
WORKDIR /tasker
ADD requirements.txt /tasker/
RUN pip install -r requirements.txt
ADD . /tasker/

docker-compose.yml:

rabbitmq:
  image: tutum/rabbitmq
  environment:
    - RABBITMQ_PASS=mypass
  ports:
    - "5672:5672"   
    - "15672:15672"
celery:
  build: .
  command: celery worker --app=tasker.tasks
  volumes:
    - .:/tasker
  links:
    - rabbitmq:rabbit

Is there something I'm missing? Why does celery exit with code 1?

Answer 1

1) install celery to python Docker image or use https://hub.docker.com/r/library/celery/

2) add full adress to rabbitmq with password in row:

celery = Celery('tasks', broker='amqp://guest:mypass@rabbit//')

Answer 2

The RabbitMQ container is not available when you are trying to bring up the celery container.

The quick and dirty way to handle this is to turn manually turn rabbitmq in the background with the -d flag first:

docker-compose up -d rabbitmq

Wait a few seconds and:

docker-compose up celery

The proper way to handle startup order is documented here -> https://docs.docker.com/compose/startup-order/

---

EDIT

I'm expanding my answer a little bit on how to accomplish this with 'wait for it'

celery:
  build: .
  command: ./wait-for-it.sh rabbitmq:5672 --timeout=2 --strict -- celery worker -A tasker.tasks
  volumes:
    - .:/tasker
  links:
    - rabbitmq

---

EDIT 2

I've also just noticed that in your compose file, the container name under the links directive is wrong. It should be just rabbitmq. This also affects the order in which your containers are being turned on.

EDIT 3

You probably need to add a hostname to the rabbitmq container as well:

rabbitmq:
  image: tutum/rabbitmq
  hostname: rabbitmq