How to send asynchronous request using flask to an endpoint with small timeout session?

Question

I am new to backend development using Flask and I am getting stuck on a confusing problem. I am trying to send data to an endpoint whose Timeout session is 3000 ms. My code for the server is as follows.

from flask import Flask, request
from gitStat import getGitStat

import requests

app = Flask(__name__)


@app.route('/', methods=['POST', 'GET'])
def handle_data():
    params = request.args["text"].split(" ")

    user_repo_path = "https://api.github.com/users/{}/repos".format(params[0])
    first_response = requests.get(
        user_repo_path, auth=(
            'Your Github Username', 'Your Github Password'))

    repo_commits_path = "https://api.github.com/repos/{}/{}/commits".format(params[
                                                                            0], params[1])
    second_response = requests.get(
        repo_commits_path, auth=(
            'Your Github Username', 'Your Github Password'))

    if(first_response.status_code == 200 and params[2] < params[3] and second_response.status_code == 200):

        values = getGitStat(params[0], params[1], params[2], params[3])

        response_url = request.args["response_url"]

        payload = {
            "response_type": "in_channel",
            "text": "Github Repo Commits Status",
            "attachments": [
                {
                    "text": values
                }
                    ]
            }

        headers = {'Content-Type': 'application/json',
                'User-Agent': 'Mozilla /5.0 (Compatible MSIE 9.0;Windows NT 6.1;WOW64; Trident/5.0)'}

        response = requests.post(response_url, json = test, headers = headers)

    else:

        return "Please enter correct details. Check if the username or reponame exists, and/or Starting date < End date. \
                Also, date format should be MM-DD"

My server code takes arguments from the request it receives and from that request's JSON object, it extracts parameters for the code. This code executes getGitStats function and sends the JSON payload as defined in the server code to the endpoint it received the request from.

My problem is that I need to send a text confirmation to the endpoint that I have received the request and data will be coming soon. The problem here is that the function, getGitStats take more than a minute to fetch and parse data from Github API.

I searched the internet and found that I need to make this call asynchronous and I can do that using queues. I tried to understand the application using RQ and RabbitMQ but I neither understood nor I was able to convert my code to an asynchronous format. Can somebody give me pointers or any idea on how can I achieve this?

Thank you.

------------Update------------

Threading was able to solve this problem. Create another thread and call the function in that thread.

Answer 1

If you are trying to have a async task in request, you have to decide whether you want the result/progress or not.

1. You don't care about the result of the task or if there where any errors while processing the task. You can just process this in a Thread and forget about the result.
2. If you just want to know about success/fail for the task. You can store the state of the task in Database and query it when needed.
3. If you want progress of the tasks like (20% done ... 40% done). You have to use something more sophisticated like celery, rabbitMQ.

For you i think option #2 fits better. You can create a simple table GitTasks.

  GitTasks
  ------------------------
  Id(PK) | Status   | Result
  ------------------------ 
   1     |Processing| -
   2     | Done     | <your result>
   3     | Error    | <error details> 

You have to create a simple Threaded object in python to processing.

 import threading  
 class AsyncGitTask(threading.Thread):
    def __init__(self, task_id, params):
        self.task_id = task_id
        self.params = params
    def run():
        ## Do processing
        ## store the result in table for id = self.task_id

You have to create another endpoint to query the status of you task.

  @app.route('/TaskStatus/<int:task_id>')
  def task_status(task_id):
      ## query GitTask table and accordingly return data

Now that we have build all the components we have to put them together in your main request.

  from Flask import url_for
  @app.route('/', methods=['POST', 'GET'])
  def handle_data():
    .....
    ## create a new row in GitTasks table, and use its PK(id) as task_id
    task_id = create_new_task_row()
    async_task = AsyncGitTask(task_id=task_id, params=params)
    async_task.start()

    task_status_url = url_for('task_status', task_id=task_id)
    ## This is request you can  return text saying 
    ## that "Your task is being processed. To see the progress 
    ## go to <task_status_url>"