Celery not running with flask application

Question

I'm using celery in my flask application but celery(3.1.8).This is my configuration with the flask application

celery.py

 from __future__ import absolute_import
 from celery import Celery
 from cuewords.settings import CELERY_BROKER_URL,CELERY_RESULT_BACKEND

 app = Celery('proj',
         broker=CELERY_BROKER_URL,
         backend=CELERY_RESULT_BACKEND)

app.conf.update(CELERY_TASK_RESULT_EXPIRES=3600)

if __name__ == '__main__':
   app.start()

setting.py

  CELERY_BROKER_URL='redis://localhost:6379/0'
  CELERY_RESULT_BACKEND='redis://localhost:6379/0'
  BROKER_TRANSPORT = 'redis'

api.py

 class Webcontent(Resource):
   def post(self,session=session):
   args = self.parser.parse_args()
   site_url = args["url"]
   url_present=Websitecontent.site_url_present(session,site_url)
    if site_url.strip() != "" and not url_present:
      try:
        #add data and commit 
        session.commit()
        websitecontent=Websitecontent(params*)
        websitecontent.update_url(id,session)
      except:
        session.rollback()
      raise
      finally:
        session.close()                
    else:
      return "No data created / data already present"

And in my model i'm adding a method to task

model.py

  from cuewords.celery import app
  class Websitecontent(Base):

    @app.task(name='update_url')
    def update_url(self,id,session):
    ...code goes here..

And this how i run the celery from command prompt

celery -A cuewords.celery worker

And i also using flower to monitor the task i can see a worker running but i couldn't see any task its empty .Any idea what im missing or doing wrong .. Thanks

Answer 1

The problem is that your tasks never get imported into the Python runtime when running the worker(s). The celery command is your entry point. And you're telling Celery to import your cuewords.celery module because thats where you're app instance resides. However, this is where the chain of events ends and no further Python code is imported.

Now, the most common mistake is to import the tasks into the same module as the Celery app instance. Unfortunately this will result in two modules trying to import things from each other and will result in a circular import error. This is no good.

To get around this one could import the task functions into the Celery app module and register them without using the decorator style. For example:

from celery import Celery
from models import my_task

app = Celery()
app.task(name='my_task')(my_task)

This would remove the need to import the app instance in your model module.

However, you're using method tasks. Method tasks need to be treated differently than function tasks as noted here: http://docs.celeryproject.org/en/latest/reference/celery.contrib.methods.html. Method tasks are different from function tasks, because they are associated with an instance of an object. In other words, the function is a class function. So to use the previous style of registering tasks, you'd need an instance of the class first. To get around this you should consider making your tasks functions instead of methods.