Retrieving result from celery worker constantly

Question

I have an web app in which I am trying to use celery to load background tasks from a database. I am currently loading the database upon request, but would like to load the tasks on an hourly interval and have them work in the background. I am using flask and am coding in python.I have redis running as well.

So far using celery I have gotten the worker to process the task and the beat to send the tasks to the worker on an interval. But I want to retrieve the results[a dataframe or query] from the worker and if the result is not ready then it should load the previous result of the worker.

Any ideas on how to do this?

Edit

I am retrieving the results from a database using sqlalchemy and I am rendering the results in a webpage. I have my homepage which has all the various links which all lead to different graphs which I want to be loaded in the background so the user does not have to wait long loading times.

Answer 1

The Celery Task is being executed by a Worker, and it's Result is being stored in the Celery Backend.

If I get you correctly, then I think you got few options:

1. Ignore the result of the graph-loading-task, store what ever you need, as a side effect of the task, in your database. When needed, query for the most recent result in that database. If the DB is Redis, you may find ZADD and ZRANGE suitable. This way you'll get the new if available, or the previous if not.

2. You can look for the result of a task if you provide it's id. You can do this when you want to find out the status, something like (where celery is the Celery app): result = celery.AsyncResult(<the task id>)

3. Use callback to update farther when new result is ready.

4. Let a background thread wait for the AsyncResult, or native_join, which is supported with Redis, and update accordingly (not recommended)

I personally used option #1 in similar cases (using MongoDB) and found it to be very maintainable and flexible. But possibly, due the nature of your UI, option #3 will more suitable for you needs.