Celery+Django -- Poll task for state and report success or failure using Django messages framework

Question

In my Django project that uses Celery (among a number of other things), I have a Celery task that will upload a file to a database in the background. I use polling to keep track of the upload progress and display a progress bar for uploading. Here are some snippets that detail the upload process:

views.py:

from .tasks import upload_task
...

upload_task.delay(datapoints, user, description) # datapoints is a list of dictionaries, user and description are simple strings

tasks.py:

from taskman.celery import app, DBTask # taskman is the name of the Django app that has celery.py
from celery import task, current_task

@task(base=DBTask)
def upload_task(datapoints, user, description):
    from utils.db.databaseinserter import insertIntoDatabase
    for count in insertIntoDatabase(datapoints, user, description):
        percent_completion = int(100 * (float(count) / float(len(datapoints))))
        current_task.update_state(state='PROGRESS', meta={'percent':percent_completion})

databaseinserter.py:

def insertIntoDatabase(datapoints, user, description):
    # iterate through the datapoints and upload them one by one
    # at the end of an iteration, yield the number of datapoints completed so far

The uploading code all works well, and the progress bar also works properly. However, I'm not sure how to send a Django message that tells the user that the upload is complete (or, in the event of an error, send a Django message informing the user of the error). When the upload begins, I do this in views.py:

from django.contrib import messages
...

messages.info(request, "Upload is in progress")

And I want to do something like this when an upload is successful:

messages.info(request, "Upload successful!")

I can't do that in views.py since the Celery task is fire and forget. Is there a way to do this in celery.py? In my DBTask class in celery.py I have on_success and on_failure defined, so would I be able to send Django messages from there?

Also, while my polling technically works, it's not currently ideal. The way the polling works currently is that it will endlessly check for a task regardless of whether one is in progress or not. It quickly floods the server console logs and I can imagine has a negative impact on performance overall. I'm pretty new to writing polling code so I'm not entirely sure of best practices and such as well as how to only poll when I need to. What is the best way to deal with the constant polling and the clogging of the server logs? Below is my code for polling.

views.py:

def poll_state(request):
    data = 'Failure'
    if request.is_ajax():
        if 'task_id' in request.POST.keys() and request.POST['task_id']:
            task_id = request.POST['task_id']
            task = AsyncResult(task_id)
            data = task.result or task.state
            if data == 'SUCCESS' or data == 'FAILURE': # not sure what to do here; what I want is to exit the function early if the current task is already completed
                return HttpResponse({}, content_type='application/json')
        else:
            data ='No task_id in the request'
            logger.info('No task_id in the request')
    else:
        data = 'Not an ajax request'
        logger.info('Not an ajax request')

    json_data = json.dumps(data)
    return HttpResponse(json_data, content_type='application/json')

And the corresponding jQuery code:

{% if task_id %}
    jQuery(document).ready(function() {
        var PollState = function(task_id) {
            jQuery.ajax({
                url: "poll_state",
                type: "POST",
                data: "task_id=" + task_id,
            }).done(function(task) {
                if (task.percent) {
                    jQuery('.bar').css({'width': task.percent + '%'});
                    jQuery('.bar').html(task.percent + '%');
                }
                else {
                    jQuery('.status').html(task);
                };
                PollState(task_id);
            });
        }
        PollState('{{ task_id }}');
    })
{% endif %}

(These last two snippets come largely from previous StackOverflow questions on Django+Celery progress bars.)

Answer 1

The simplest answer to reduce logging and overhead is to put a timeout on your next PollState call. The way your function is written right now it immediately polls again. Something simple like:

setTimeout(function () { PollState(task_id); }, 5000);

This will drastically reduce your logging issue and overhead.

---

Regarding your Django messaging question you'd need to pull those completed tasks out with some sort of processing. One way to do it is a Notification model—or similar—which you can then add a piece of middleware to fetch unread notifications and inject them into the messages framework.