Making Celery returning a Future

Question

I am using the tornado web framework. Is it possible to make a celery task returning an object of the Future class, in order to use it in an @gen.coroutine decorated handler?

What I am trying to do is the following:

    class TornadoRequestHandler(BaseHandler):

         @gen.coroutine
         def get(self):
              result = yield celery_task.apply_aync()
              self.write(result)
              self.finish()

I've seen tornado-celery, but that isn't exactly what i am trying to achieve.

Answer 1

As far as I know, the only way to do this is via tornado-celery, which will let you do this:

class TornadoRequestHandler(BaseHandler):

     @gen.coroutine
     def get(self):
          result = yield gen.Task(celery_task.apply_aync)
          self.write(result)
          self.finish()

The reason is that the behavior you want when using gen.coroutine relies on all the async methods it calls to take a callback kwarg, which gets called when the method completes. celery_task.apply_async doesn't take a callback kwarg, so gen.coroutine can't be used directly. It looks like tornado-celery works by exploiting the fact that apply_async does take an **options parameter, which can be any arbitrary kwargs. This means that apply_async actually will accept a callback kwarg, but just ignore it. tornado-celery takes advantage of this by overriding the class in Celery responsible for publishing tasks, and changes the publishing process to publish your task, and then consume from the result queue that gets published to when the tasks completes. The consuming code executes the function provided by the normally ignored callback kwarg, which it pulls out **options.

I'm not sure how clear that explanation, was, but the tl;dr version is tornado-celery provides the simplest way of getting as close possible to the behavior you want.