How can I manage a queue of requests in my Flask service?

Question

I am using flask to get POST requests from the users and process the data without returning any response.

To process the user requests the solution that I currently have is:

def process_data(request_data):
    # do_the_processing will process the data 
    do_the_processing(request_data)
    # Terminate the spawned process
    sys.exit(0)

@app.route('/', methods =['POST'])
def index():
    request_data = request.get_json()
    p = Process(target=process_data, args=(request_data,))
    p.start()
    return '200'

But this will spawn a new process for every user request. What if 1000 users make a request to the service at the same time. OS won't be able to handle these many processes. I am ready to make the users wait until their turn comes in the queue. do_the_processing method uses another web API to process the data and then write files in the memory.

I need to maintain a queue in which I can put the request as it arrives and then process the request_data on first come first serve basis.

To increase the processing we can use 4 processes to get the data from the queue and process it. Once the data has been processed we'll fetch another data from the queue and process it but the catch is we should use only 4 processes to do the processing.

The current solution will spawn 1000 processes if 1000 users make a request to the server.

How can I implement this type of functionality, any ideas?

Answer 1

I've never found a good way of using the built in queue for sharing a work queue between processes. By that I mean gunicorn/uwsgi processes not manager/workers.

My suggestion would be to use something like redis or rabbit mq for distributing the workload. There are several frameworks for this such as celery or just using pika. This would build your application scaleable in a way. Beging able to distribute API, work queue and workers on separate instances in what scale you require.