Updating client that a celery task has finished

Question

I have the following requirement. I am creating a web app (no frontend) that will receive post and get requests. It's functionality will be the following. Client will make a post on a URL triggering a large and time expensive task. A Client will also have the ability to make a Get request and retrieve the results of the taks.

I was thinking of using Celery to start the task. The task is a very large arithmetic computation. My problem is on how to notify the client that the task he triggered has finished and he can retrieve the result.

So I was thinking it like this:

client ->posts to web app -> web app starts task to celery -> request closed

web app notifies -> client triggered task  # what is the proper way

client -> get request to web app -> web app returns result

I know there isn't one proper way, so I' ll be a bit more specific. For the time being the task will be triggered by another flask application, automatically. Is there a way the flask application to be automatically notified that the task is finished and get the results and store them in its own database? And a second question what would be the most generic solution that would work even if the client is a flask app, an android app an IOS app etc.

The reason I also ask is because I had a debate on whether an api post the results back to the client or the client should be responsible to check and get the results.

Answer 1

I've had to make a similar decision in the past, where a client needed to be updated upon completion of a long-running celery task. There are many different ways to design the system to solve the problem:

1. Status Endpoint with Polling - This is the simplest solution. Expose an endpoint in flask which returns the status of a job and simply poll from the client. There are several things to be mindful of however, namely the number of clients and frequency of polling. If you have too many clients or are polling too frequently, you could run into performance issues. If you have a few clients and don't need instant status updates, this is ideal.
2. WebSockets - Open up a WebSocket between the client and flask server, and use this to push updates when your task is complete. This approach will scale better with multiple clients and will allow for near instant notifications.
3. Callback URL - You mentioned that the client was a Flask app. You could expose a callback route on the client server and provide this to celery when starting the task. Upon task completion, Celery can POST to the callback URL and this will notify the client.

As for the most generic solution, that would be either the status endpoint or WebSockets - Android & iOS have HTTP functionality built in and can handle WebSockets using libraries. The callback URL solution on the other hand is specific to Flask.