how to call a url asynchronously from a celery task

Question

I'm using celery with Tornado and I was wondering how can I call a url asynchronously inside a task.

I'm looking for something in the lines of:

@celery.task
def my_task(data):
    def handle_response(response):
        if response.error:
            print "error"
        else:
            print "success"

    http_client = httpclient.AsyncHTTPClient()
    http_client.fetch('some url', handle_response, method='POST', body=data)

or:

@celery.task
@gen.coroutine
def my_task(data):
    http_client = httpclient.AsyncHTTPClient()
    response = yield http_client.fetch('some url', method='POST', body=data)
    raise gen.Result(response.body)

My problem now is that I don't get to the response handler. Using HttpClient instead works but since it blocks the server, I'm looking for a non-blocking solution.

BTW, My broker is redis and I wish to keep it (tornado-celery callbacks work only with pika if it provides a solution)

Answer 1

In my opinion your approach is an overhead. Celery is already made to execute jobs asynchronously so what better place that a celery task to do a blocking URL call ? Adding an async url call in async task is a kind of overhead. I hope this helps.