Nginx upstream prematurely closed connection while reading response header from upstream, for large requests

Question

I am using nginx and node server to serve update requests. I get a gateway timeout when I request an update on large data. I saw this error from the nginx error logs :

2016/04/07 00:46:04 [error] 28599#0: *1 upstream prematurely closed connection while reading response header from upstream, client: 10.0.2.77, server: gis.oneconcern.com, request: "GET /update_mbtiles/atlas19891018000415 HTTP/1.1", upstream: "http://127.0.0.1:7777/update_mbtiles/atlas19891018000415", host: "gis.oneconcern.com"

I googled for the error and tried everything I could, but I still get the error.

My nginx conf has these proxy settings:

    ##
    # Proxy settings
    ##

    proxy_connect_timeout 1000;
    proxy_send_timeout 1000;
    proxy_read_timeout 1000;
    send_timeout 1000;

This is how my server is configured

server {
listen 80;

server_name gis.oneconcern.com;
access_log /home/ubuntu/Tilelive-Server/logs/nginx_access.log;
error_log /home/ubuntu/Tilelive-Server/logs/nginx_error.log;

large_client_header_buffers 8 32k;
location / {
    proxy_pass http://127.0.0.1:7777;
    proxy_redirect off;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $http_host;
    proxy_cache_bypass $http_upgrade;
}

location /faults {
    proxy_pass http://127.0.0.1:8888;
    proxy_http_version 1.1;
    proxy_buffers 8 64k;
    proxy_buffer_size 128k;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
}

}

I am using a nodejs backend to serve the requests on an aws server. The gateway error shows up only when the update takes a long time (about 3-4 minutes). I do not get any error for smaller updates. Any help will be highly appreciated.

Node js code :

app.get("/update_mbtiles/:earthquake", function(req, res){
var earthquake = req.params.earthquake
var command = spawn(__dirname + '/update_mbtiles.sh', [ earthquake, pg_details ]);
//var output  = [];

command.stdout.on('data', function(chunk) {
//    logger.info(chunk.toString());
//     output.push(chunk.toString());
});

command.stderr.on('data', function(chunk) {
  //  logger.error(chunk.toString());
 //   output.push(chunk.toString());
});

command.on('close', function(code) {
    if (code === 0) {
        logger.info("updating mbtiles successful for " + earthquake);
        tilelive_reload_and_switch_source(earthquake);
        res.send("Completed updating!");
    }
    else {
        logger.error("Error occured while updating " + earthquake);
        res.status(500);
        res.send("Error occured while updating " + earthquake);
    }
});
});

function tilelive_reload_and_switch_source(earthquake_unique_id) {
tilelive.load('mbtiles:///'+__dirname+'/mbtiles/tipp_out_'+ earthquake_unique_id + '.mbtiles', function(err, source) {
    if (err) {
        logger.error(err.message);
        throw err;
    }
    sources.set(earthquake_unique_id, source); 
    logger.info('Updated source! New tiles!');
});
}

Thank you.

Answer 1

I solved this by setting a higher timeout value for the proxy:

location / {
    proxy_read_timeout 300s;
    proxy_connect_timeout 75s;
    proxy_pass http://localhost:3000;
}

Documentation: https://nginx.org/en/docs/http/ngx_http_proxy_module.html

Answer 2

I think that error from Nginx is indicating that the connection was closed by your nodejs server (i.e., "upstream"). How is nodejs configured?

Answer 3

I had the same error for quite a while, and here what fixed it for me.

I simply declared in service that i use what follows:

Description= Your node service description
After=network.target

[Service]
Type=forking
PIDFile=/tmp/node_pid_name.pid
Restart=on-failure
KillSignal=SIGQUIT
WorkingDirectory=/path/to/node/app/root/directory
ExecStart=/path/to/node /path/to/server.js

[Install]
WantedBy=multi-user.target

What should catch your attention here is "After=network.target". I spent days and days looking for fixes on nginx side, while the problem was just that. To be sure, stop running the node service you have, launch the ExecStart command directly and try to reproduce the bug. If it doesn't pop, it just means that your service has a problem. At least this is how i found my answer.

For everybody else, good luck!

Answer 4

I stumbled upon *145660 upstream prematurely closed connection while reading upstream Nginx error log entry when trying to download a 2GB file from the server Nginx was a proxy for. The message indicates that the "upstream" closed connection, but in fact it was related to proxy_max_temp_file_size setting:

Syntax: proxy_max_temp_file_size size;
Default: proxy_max_temp_file_size 1024m;
Context: http, server, location

When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file. The size of data written to the temporary file at a time is set by the proxy_temp_file_write_size directive.

The zero value disables buffering of responses to temporary files.

This restriction does not apply to responses that will be cached or stored on disk.

The symptoms:

• download was being forcibly stopped at around 1GB,
• Nginx claimed that upstream closed connection, but without proxy server was returning the full content.

The solution:

• increased proxy_max_temp_file_size for proxied location to 4096m and it started sending full content.