flask application with background threads

Question

I am creating a flask application, for one request I need to run some long running job which is not required to wait on the UI. I will create a thread and send a message to UI. The thread will calculate and update the database. But, UI will see a message upon submit. Below is my implementation, but it is running the thread and then sending the output to UI which is not I prefer. How can I run this thread in the background?

@app.route('/someJob')
def index():
    t1 = threading.Thread(target=long_running_job)
    t1.start()
    return 'Scheduled a job'

def long_running_job
    #some long running processing here

How can I make thread t1 to run the background and immediately send message in return?

Answer 1

Try this example, tested on Python 3.4.3 / Flask 0.11.1

from flask import Flask
from time import sleep
from concurrent.futures import ThreadPoolExecutor

# DOCS https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
executor = ThreadPoolExecutor(2)

app = Flask(__name__)


@app.route('/jobs')
def run_jobs():
    executor.submit(some_long_task1)
    executor.submit(some_long_task2, 'hello', 123)
    return 'Two jobs were launched in background!'


def some_long_task1():
    print("Task #1 started!")
    sleep(10)
    print("Task #1 is done!")


def some_long_task2(arg1, arg2):
    print("Task #2 started with args: %s %s!" % (arg1, arg2))
    sleep(5)
    print("Task #2 is done!")


if __name__ == '__main__':
    app.run()

Answer 2

Check out Flask-Executor which uses concurrent.futures in the background and makes your life very easy.

from flask_executor import Executor

executor = Executor(app)

@app.route('/someJob')
def index():
    executor.submit(long_running_job)
    return 'Scheduled a job'

def long_running_job
    #some long running processing here

This not only runs jobs in the background but gives them access to the app context. It also provides a way to store jobs so users can check back in to get statuses.

Answer 3

The best thing to do for stuff like this is use a message broker. There is some excellent software in the python world meant for doing just this:

• Celery (http://www.celeryproject.org/), and
• RQ (http://python-rq.org/).

Both are excellent choices.

It's almost never a good idea to spawn a thread the way you're doing it, as this can cause issues processing incoming requests, among other things.

If you take a look at the celery or RQ getting started guides, they'll walk you through doing this the proper way!