Upstream prematurely closed connection while reading response header from upstream Ruby project

Question

I am trying to reverse proxies a ruby project on GCP with NGINX, my /etc/nginx/sites-available/default file looks like this

 server {
     large_client_header_buffers 4 16k;
     listen 80 default_server;
     #server_name my-devops-staging.com  
     listen [::]:80 default_server;
     #return 301 https://$host$request_uri;
     # SSL configuration
     #
     listen 443 ssl default_server;
     listen [::]:443 ssl default_server;
     ssl_certificate           /etc/nginx/certificate.crt;
     ssl_certificate_key       /etc/nginx/key.key;
     ssl off;
     ssl_session_cache  builtin:1000  shared:SSL:10m;
     ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
     ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
     ssl_prefer_server_ciphers on;
     # Note: You should disable gzip for SSL traffic.
     # See: https://bugs.debian.org/773332
     #
     # Read up on ssl_ciphers to ensure a secure configuration.
     # See: https://bugs.debian.org/765782
     #
     # Self signed certs generated by the ssl-cert package
     # Don't use them in a production server!
     #
     # include snippets/snakeoil.conf;
     root /var/www/html;
     # Add index.php to the list if you are using PHP
     index index.html index.htm index.nginx-debian.html;
     location / {
         proxy_set_header        Host $server_name;
         proxy_set_header        X-Real-IP $remote_addr;
         proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header        X-Forwarded-Proto $scheme;
         # Fix the “It appears that your reverse proxy set up is broken" error.
         proxy_pass http://localhost:8080;
         # proxy_read_timeout 300;
         proxy_read_timeout  9000;
         proxy_request_buffering off;
         proxy_buffering off;
         proxy_redirect off;
     }
}

What could I be doing wrong, whenever I run

$ sudo service nginx restart

I get these errors in the error.log

2018/03/27 08:32:50 [error] 2959#2959: *64 upstream prematurely closed connection while reading response header from upstream, client: 130.211.2.175, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "my-devops-staging.com" 
2018/03/27 08:32:53 [error] 2959#2959: *66 upstream prematurely closed connection while reading response header from upstream, client 130.211.2.87, server: , request: "GET / HTTP/1.1", upstream: http://127.0.0.1:8080/", host: "my-devops-staging.com"

Answer 1

Make the communication between your proxy and backend more loyal by adding these params to your proxy Nginx config file:

location / {

    proxy_http_version 1.1;  # you need to set this in order to use params below.

    proxy_temp_file_write_size 64k;
    proxy_connect_timeout 10080s;
    proxy_send_timeout 10080;
    proxy_read_timeout 10080;
    proxy_buffer_size 64k;
    proxy_buffers 16 32k;
    proxy_busy_buffers_size 64k;
    proxy_redirect off;
    proxy_request_buffering off;
    proxy_buffering off;
    proxy_pass <whatever_here>;
}

The magic numbers I've took from the production environment that works for us. You may want to consider changing those numbers to fit your environment and number of connections, etc.

I hope this helps.

Answer 2

You are actually timing out in connecting into your Rails application, you can lower that proxy_read_timeout back to 300 and up your proxy_connect_timeout to match it, or higher if it keeps happening. Just add those lines in your location \ {...} block:

 location / {
     proxy_set_header        Host $server_name;
     proxy_set_header        X-Real-IP $remote_addr;
     proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header        X-Forwarded-Proto $scheme;
     proxy_pass http://localhost:8080;
     proxy_read_timeout 300; # Reducing this
     proxy_connect_timeout 300;  # Adding this
     proxy_request_buffering off;
     proxy_buffering off;
     proxy_redirect off;
 }