couldn't start Celery with docker-compose

Question

I have Flask app with Celery worker and Redis and it's working normally as expected when running on local machine. Then I tried to Dockerize the application. When I trying to build/start the services ( ie, flask app, Celery, and Redis) using sudo docker-compose up all services are running except Celery and showing an error as

ImportError: No module named 'my_celery'

But, the same code working in local machine without any errors. Can any one suggest the solution?

Dockerfile

FROM python:3.5-slim
WORKDIR celery_sample
ADD . /celery_sample
RUN pip install -r requirements.txt
EXPOSE 8000

docker-compose.yml

version: "3"
services:

  web:
    build:
      context: .
      dockerfile: Dockerfile
    command: "python my_celery.py"
    ports:
      - "8000:8000"
    networks:
      - webnet
    volumes:
      - .:/celery_sample

  redis:
    image: redis
    networks:
      - webnet


  celery:
    image: celery:3.1.25
    command: "celery worker -A my_celery -l INFO"
    volumes:
      - .:/celery_sample
    networks:
      - webnet

networks:
  webnet:

requirements.txt

flask==0.10
redis
requests==2.11.1
celery==3.1.25

my_celery.py ( kindly ignore the logic)

from flask import Flask
from celery import Celery
flask_app = Flask(__name__)
celery_app = Celery('my_celery')
celery_app.config_from_object('celeryconfig')



@celery_app.task
def add_celery():
    return str(int(10)+int(40))


@flask_app.route('/')
def index():
    return "Index Page"

@flask_app.route('/add')
def add_api():
    add_celery.delay()
    return "Added to Queue"

if __name__ == '__main__':
    flask_app.debug = True
    flask_app.run(host='0.0.0.0', port=8000)

celeryconfig.py

## Broker settings.
BROKER_URL = 'redis://localhost:6379/0'    
## Using the database to store task state and results.
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

Answer 1

First of all the celery image is deprecated in favour of standard python image more info here.

WORKDIR sets the working directory for all the command after it is defined in the Dockerfile, which means the command which you are try to run will run from that directory. Docker image for celery sets the working directory to /home/user.

Since your code is mounted on /celery_smaple and the working directory is /home/user, Celery is not able to find your python module.

One alternative is to cd into the mounted directory and execute the command:

celery:
    image: celery:3.1.25
    command: "cd /celery_sample && celery worker -A my_celery -l INFO"
    volumes:
      - .:/celery_sample
    networks:
      - webnet

notice the command

And another one is to create your own image with WORKDIR set to /celery_sample eg:

FROM python:3.5
RUN pip install celery==3.1.25
WORKDIR /celery_sample

after building you own image you can use the compose file by changing the image of celery service

Edit

You need to link the services to one another in order to communicate:

version: "3"
services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    command: "python my_celery.py"
    ports:
      - "8000:8000"
    networks:
      - webnet
    volumes:
      - .:/celery_sample
    links:
      - redis

  redis:
    image: redis
    networks:
      - webnet

  celery:
    image: celery:3.1.25
    command: "celery worker -A my_celery -l INFO"
    volumes:
      - .:/home/user
    networks:
      - webnet
    links:
      - redis

networks:
  webnet:

and your configuration file should be:

## Broker settings.
BROKER_URL = 'redis://redis:6379/0'    
## Using the database to store task state and results.
CELERY_RESULT_BACKEND = 'redis://redis:6379/0'

once you have linked the services in compose file you can access the service by using the service name as the hostname.

Answer 2

The simplest change you can make is to map the directory for celery image correctly in yaml

  celery:
    image: celery:3.1.25
    command: "celery worker -A my_celery -l INFO"
    volumes:
      - .:/home/user/
    networks:
      - webnet

Also as @vedarthk pointed, you should be using the official Python image for this instead of celery image