Nginx cuts off static files downloads early

Question

I have a Flask app that redirects requests that should get served static files to NGINX through x-accel-redirect. On occasion, those downloads will get cut off before being finished. For example, through cURL, I'd see:

curl http://my_server/some_static_file.tar > temp.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 77 14.4G   77 11.2G    0     0  55.8M      0  0:04:24  0:03:25  0:00:59 58.9M
curl: (18) transfer closed with 3449105332 bytes remaining to read

This seems to happen more often with very big files (10gb+), but I've seen it also happen on smaller files of ~90mb. Nginx access logs show requests coming through and being served different, incomplete amounts of data:

1.2.3.4 - - [18/Apr/2017:01:16:26 +0000] "GET /some/flask/static/file/path HTTP/1.1" 200 15146008576 "-" "curl/7.38.0" "5.6.7.8"
1.2.3.5 - - [18/Apr/2017:01:16:29 +0000] "GET /some/flask/static/file/path HTTP/1.1" 200 15441739776 "-" "curl/7.38.0" "6.7.8.9"

errors.log has nothing useful.

My relevant flask config is as follows:

response = make_response('')
response.headers.set('X-Accel-Redirect', '/_special_nginx_path/' + file_name)
response.headers.set('Content-Disposition', 'attachment',
                     filename=file_name)
# have tried both with and without setting content-length
response.headers.set('Content-Length', os.path.getsize(file_path))
try:
    response.mimetype = mimetypes.guess_type(file_name)[0]
    if not response.mimetype:
        response.mimetype = 'application/octet-stream'
except AttributeError:
    response.mimetype = 'application/octet-stream'
return response

My relevant NGINX config is as follows (where a uWSGI server running my flask app is running at 127.0.0.1:1234):

location / {
            proxy_pass http://127.0.0.1:1234;
            proxy_redirect off;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }



location /_special_nginx_path {
           internal;
           alias /path/to/static/files;
    }

Answer 1

Please check your disk usage, it may happen due to this, check the nginx error logs first, the error log may have logs like:

2018/10/28 14:20:24 [crit] 5432#5432: *75 pwritev() "/var/lib/nginx/uwsgi/1/00/0000000001" failed (28: No space left on device) while reading upstream,

First, identify which partition doesn’t have free space. You can do so by typing the following command in the terminal:

df -h

You’ll now see the following details on the screen:

File system. Size. Used. Available. Used. Mounted on.

Go through the partition details and check whether any partition’s disk space usage has reached up to 100%.

Once you find the partition, open it and delete useless files so as to free up the disk space and fix the problem.

In case the partition is mounted on the system memory (indicated by the TMPFS directory), run the below command to unmount it.

Umount path_to_the_directory.

Now, restart Nginx. The error will now disappear from the file.

To prevent the no space left on device error in future, edit the Nginx configuration file (or your website’s config file) and increase the value of the key zone.

Users face the problem because they configure the OS to serve cache files from RAM. Although this can boost the performance of your site quickly, it reduces the amount of RAM available for other applications running on the server and leads to out of memory error.

If your server uses SSD instead of HDD, you don’t have to mount the partition into the system memory.

Thanks to the blog which helped me...