Django long request timeout

Question

I have a view in a Django application that take as input a large csv file, it cycle all the rows and insert datas on DB. Locally I have no problem but when I deploy my proj online the view give me back a timeout after some time. My webserver configuration is Nginx + Gunicorn. I tried to increase timeout on Gunicorn and proxy_connect_timeout, proxy_read_timeout on Nginx by writing a large number (120000) and now is better, I get timeout after about 90 seconds instead of 30 (default for Gunicorn) but still is not what I expected, and it's not enough to finish my job. Also I don't like so much this approach, I don't want infinite timeout for every request. What's the best approach to deal with long request and not have timeout? Maybe by answering to user with a message and then run the job in background? Any suggestion? Thanks.

Answer 1

Using Celery With Django for Background Task Processing that means process the CSV file with celery async task.

OR

As a quick hack, if you don't want to use celery; use multi-threading to process the CSV and save the result of the processing in DB or file and server the result from DB or file.

Note: Never Process big files on main thread; always try to use a different server to process the big files.If different server is not possible then try to process it in background task

Many solutions can be found on this StackOverflow link

Answer 2

    I spend more time to increase the request timeout. And Finally I got the soluton
    For **Django Rest API ( Nginx + supervisor + Gunicorn + AWS )**

     1. Add timeout in the gunicorn(Supervisor config)

        [program:gunicorn]
         command = <env_path>/env/bin/gunicorn --pythonpath=<python_path> --name=<nginx_name> --bind unix:/tmp/myproject.sock --workers 3 --graceful-timeout=900 --timeout=900 <project_path>.wsgi:application
        directory=<project_path>
        autostart=true
        autorestart=true
        stopsignal=TERM
        stdout_logfile=/var/log/gunicorn.stdout.log
        stdout_logfile_maxbytes=1MB
        stdout_logfile_backups=5
        stderr_logfile=/var/log/gunicorn.stderr.log
        stderr_logfile_maxbytes=1MB
        stderr_logfile_backups=5

     2. Add proxy on nginx file

     proxy_connect_timeout       900;
     proxy_send_timeout          900;
     proxy_read_timeout          900;
     send_timeout                900;

     3. Changes in Load Balancer ( If you configure  -> in EC2 instance)
 [Load Balancer][1]


  [1]: https://i.stack.imgur.com/uSUrK.png