Show a progress bar for my multithreaded process

Question

I have a simple Flask web app that make many HTTP requests to an external service when a user push a button. On the client side I have an angularjs app.

The server side of the code look like this (using multiprocessing.dummy):

worker = MyWorkerClass()
pool = Pool(processes=10)
result_objs = [pool.apply_async(worker.do_work, (q,))
                           for q in queries]
pool.close() # Close pool
pool.join()  # Wait for all task to finish
errors = not all(obj.successful() for obj in result_objs)
# extract result only from successful task
items = [obj.get() for obj in result_objs if obj.successful()]

As you can see I'm using apply_async because I want to later inspect each task and extract from them the result only if the task didn't raise any exception.

I understood that in order to show a progress bar on client side, I need to publish somewhere the number of completed tasks so I made a simple view like this:

@app.route('/api/v1.0/progress', methods=['GET'])
def view_progress():
    return jsonify(dict(progress=session['progress']))

That will show the content of a session variable. Now, during the process, I need to update that variable with the number of completed tasks (the total number of tasks to complete is fixed and known).

Any ideas about how to do that? I working in the right direction?

I'have seen similar questions on SO like this one but I'm not able to adapt the answer to my case.

Thank you.

Answer 1

For interprocess communication you can use a multiprocessiong.Queue and your workers can put_nowait tuples with progress information on it while doing their work. Your main process can update whatever your view_progress is reading until all results are ready.

A bit like in this example usage of a Queue, with a few adjustments:

In the writers (workers) I'd use put_nowait instead of put because working is more important than waiting to report that you are working (but perhaps you judge otherwise and decide that informing the user is part of the task and should never be skipped).

The example just puts strings on the queue, I'd use collections.namedtuples for more structured messages. On tasks with many steps, this enables you to raise the resolution of you progress report, and report more to the user.