Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

upstream timed out (110: Connection timed out) for static content?

Tags:

I've a situation where two webservers are setup with nginx as loadbalancer and are backends themselves. Distribution is Debian Wheezy. Config is the same on both servers (Quad-Core with 32GB RAM)

TCP

#/etc/sysctl.conf
vm.swappiness=0
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_sack=1
net.ipv4.ip_local_port_range=2000 65535
net.ipv4.tcp_max_syn_backlog=65535
net.core.somaxconn=65535
net.ipv4.tcp_max_tw_buckets=2000000
net.core.netdev_max_backlog=65535
net.ipv4.tcp_rfc1337=1
net.ipv4.tcp_fin_timeout=5
net.ipv4.tcp_keepalive_intvl=15
net.ipv4.tcp_keepalive_probes=5
net.core.rmem_default=8388608
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 16384 16777216
net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_tw_reuse=1
fs.file-max=3000000

Nginx

#/etc/nginx/nginx.conf
user www-data www-data;
worker_processes 8;
worker_rlimit_nofile 300000;
pid /run/nginx.pid;

events {
        worker_connections 8192;
        use epoll;
        #multi_accept on;
}
http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 10;
        types_hash_max_size 2048;
        server_tokens off;

        open_file_cache max=200000 inactive=20s;
        open_file_cache_valid 30s;
        open_file_cache_min_uses 5;
        open_file_cache_errors on;

        gzip on;
        gzip_vary on;
        gzip_proxied any;
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
        gzip_min_length 10240;
        gzip_disable "MSIE [1-6]\.";
}

server {
    listen <PUBLIC-IPv4>:8080 default_server;
    listen <PUBLIC-IPv6>:8080 default_server;
    listen 127.0.0.1:8080 default_server;
    listen [::1]:8080 default_server;
    server_name backend01.example.com;
    access_log /var/log/nginx/access upstream;
    error_log /var/log/nginx/error;

    root /var/www/project/web;
    index app.php;
    error_page 500 501 502 503 504 505 /50x.html;
    client_max_body_size 8m;

    location ~ /\. { return 403; }
    try_files $uri $uri/ /app.php?$query_string;
    location ~ ^/(config|app_dev|app)\.php(/|$) {
        include fastcgi_params;
        # fastcgi_split_path_info ^(.+\.php)(/.*)$;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_pass_header Authorization;
        fastcgi_buffers 16 16k;
        fastcgi_buffer_size 32k;
        fastcgi_param HTTPS on;
    }
}

upstream www {
    ip_hash;
    server [::1]:8080;
    server backend02:8080;
}

server {
    listen <LOADBALANCER-IPv4>:443 ssl spdy;
    server_name www.example.com;
    access_log /var/log/nginx/access main;
    error_log /var/log/nginx/error;

    ssl                  on;
    ssl_certificate      /etc/ssl/example.com.crt;
    ssl_certificate_key  /etc/ssl/example.com.key;
    ssl_protocols        TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers          ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
    ssl_session_cache    shared:SSL:20m;
    ssl_session_timeout  10m;

    root /var/www/project/web;
    error_page 500 501 502 503 504 505 /50x.html;
    client_max_body_size 8m;

    location /00_templates { return 403; }
    location / {
        proxy_read_timeout 300;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://www;
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
        proxy_buffers 16 16k;
        proxy_buffer_size 32k;
    }
}

When simulating connections from 3 clients with

ab -c 200 -n 40000 -q https://www.example.com/static/file.html

Why do I get

upstream timed out (110: Connection timed out) while connecting to upstream

in nginx log? An upstream timeout for 600 concurrent connections for a static file!? While running ab test I can see on first backend node:

# netstat -tan | grep ':8080 ' | awk '{print $6}' | sort | uniq -c
      2 LISTEN
     55 SYN_SENT
  37346 TIME_WAIT
like image 839
rabudde Avatar asked Feb 05 '15 14:02

rabudde


People also ask

What is upstream timeout?

The upstream timeout error generally triggers when the upstream takes too much to answer the request and NGINX thinks the upstream already failed in processing the request. [ 0] proxy_read_timeout – Defines a timeout for reading a response from the proxied server.

What is timed out in connection?

A server connection timeout means that a server is taking too long to reply to a data request made from another device.

Why does Nginx timeout when reading response header from upstream?

upstream timed out (110: Connection timed out) while reading response header from upstream Probably because Nginx tries to get the data from puma again and again.The funny thing is that the error caused the timeout message even if I'm calling a different action in the controller, so, a single typo blocks all the app.

Why does Nginx keep disconnecting from upstream server?

This caused upstream to sometimes fall in timeout and drop the connection, while nginx didn't understand why. Raising the upstream server value to match nginx' one solved the issue.

How many time_wait connections in backend now?

That's it and thousand of TIME_WAIT connections in backend are gone, there are only round about 150 now instead of 30-40k. Would you happen to know what response code is sent back to the client in case the upstream timed out error comes?

How long Nginx will wait for upstream app to return?

Same thing happened to me and I used 1 hour timeout for an internal app at work: With this, NGINX will wait for an hour (3600s) for its upstream to return something. Show activity on this post. You should always refrain from increasing the timeouts, I doubt your backend server response time is the issue here in any case.


1 Answers

OK, I didn't like reading manuals, but to answer my question:

nginx close upstream connection after request

solved it. So what was the problem: I've configured upstream to use keepalive but Nginx doc suggests to set following options in proxy location:

    proxy_http_version 1.1;
    proxy_set_header Connection "";

That's it and thousand of TIME_WAIT connections in backend are gone, there are only round about 150 now instead of 30-40k.

like image 103
rabudde Avatar answered Sep 25 '22 12:09

rabudde