Tornado celery integration hacks

Question

Since nobody provided a solution to this post plus the fact that I desperately need a workaround, here is my situation and some abstract solutions/ideas for debate.

My stack:

1. Tornado
2. Celery
3. MongoDB
4. Redis
5. RabbitMQ

My problem: Find a way for Tornado to dispatch a celery task ( solved ) and then asynchronously gather the result ( any ideas? ).

Scenario 1: (request/response hack plus webhook)

• Tornado receives a (user)request, then saves in local memory (or in Redis) a { jobID : (user)request} to remember where to propagate the response, and fires a celery task with jobID
• When celery completes the task, it performs a webhook at some url and tells tornado that this jobID has finished ( plus the results )
• Tornado retrieves the (user)request and forwards a response to the (user)

Can this happen? Does it have any logic?

Scenario 2: (tornado plus long-polling)

• Tornado dispatches the celery task and returns some primary json data to the client (jQuery)
• jQuery does some long-polling upon receipt of the primary json, say, every x microseconds, and tornado replies according to some database flag. When the celery task completes, this database flag is set to True, then jQuery "loop" is finished.

Is this efficient?

Any other ideas/schemas?

Answer 1

My solution involves polling from tornado to celery:

class CeleryHandler(tornado.web.RequestHandlerr):

    @tornado.web.asynchronous
    def get(self):    

        task = yourCeleryTask.delay(**kwargs)

        def check_celery_task():
            if task.ready():
                self.write({'success':True} )
                self.set_header("Content-Type", "application/json")  
                self.finish()
            else:   
                tornado.ioloop.IOLoop.instance().add_timeout(datetime.timedelta(0.00001), check_celery_task)

        tornado.ioloop.IOLoop.instance().add_timeout(datetime.timedelta(0.00001), check_celery_task)

Here is post about it.