I've a situation where two webservers are setup with nginx as loadbalancer and are backends themselves. Distribution is Debian Wheezy. Config is the same on both servers (Quad-Core with 32GB RAM)
TCP
#/etc/sysctl.conf
vm.swappiness=0
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_sack=1
net.ipv4.ip_local_port_range=2000 65535
net.ipv4.tcp_max_syn_backlog=65535
net.core.somaxconn=65535
net.ipv4.tcp_max_tw_buckets=2000000
net.core.netdev_max_backlog=65535
net.ipv4.tcp_rfc1337=1
net.ipv4.tcp_fin_timeout=5
net.ipv4.tcp_keepalive_intvl=15
net.ipv4.tcp_keepalive_probes=5
net.core.rmem_default=8388608
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 16384 16777216
net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_tw_reuse=1
fs.file-max=3000000
Nginx
#/etc/nginx/nginx.conf
user www-data www-data;
worker_processes 8;
worker_rlimit_nofile 300000;
pid /run/nginx.pid;
events {
worker_connections 8192;
use epoll;
#multi_accept on;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 10;
types_hash_max_size 2048;
server_tokens off;
open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 5;
open_file_cache_errors on;
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
gzip_min_length 10240;
gzip_disable "MSIE [1-6]\.";
}
server {
listen <PUBLIC-IPv4>:8080 default_server;
listen <PUBLIC-IPv6>:8080 default_server;
listen 127.0.0.1:8080 default_server;
listen [::1]:8080 default_server;
server_name backend01.example.com;
access_log /var/log/nginx/access upstream;
error_log /var/log/nginx/error;
root /var/www/project/web;
index app.php;
error_page 500 501 502 503 504 505 /50x.html;
client_max_body_size 8m;
location ~ /\. { return 403; }
try_files $uri $uri/ /app.php?$query_string;
location ~ ^/(config|app_dev|app)\.php(/|$) {
include fastcgi_params;
# fastcgi_split_path_info ^(.+\.php)(/.*)$;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass_header Authorization;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
fastcgi_param HTTPS on;
}
}
upstream www {
ip_hash;
server [::1]:8080;
server backend02:8080;
}
server {
listen <LOADBALANCER-IPv4>:443 ssl spdy;
server_name www.example.com;
access_log /var/log/nginx/access main;
error_log /var/log/nginx/error;
ssl on;
ssl_certificate /etc/ssl/example.com.crt;
ssl_certificate_key /etc/ssl/example.com.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
ssl_session_cache shared:SSL:20m;
ssl_session_timeout 10m;
root /var/www/project/web;
error_page 500 501 502 503 504 505 /50x.html;
client_max_body_size 8m;
location /00_templates { return 403; }
location / {
proxy_read_timeout 300;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://www;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_buffers 16 16k;
proxy_buffer_size 32k;
}
}
When simulating connections from 3 clients with
ab -c 200 -n 40000 -q https://www.example.com/static/file.html
Why do I get
upstream timed out (110: Connection timed out) while connecting to upstream
in nginx log? An upstream timeout for 600 concurrent connections for a static file!?
While running ab
test I can see on first backend node:
# netstat -tan | grep ':8080 ' | awk '{print $6}' | sort | uniq -c
2 LISTEN
55 SYN_SENT
37346 TIME_WAIT
The upstream timeout error generally triggers when the upstream takes too much to answer the request and NGINX thinks the upstream already failed in processing the request. [ 0] proxy_read_timeout – Defines a timeout for reading a response from the proxied server.
A server connection timeout means that a server is taking too long to reply to a data request made from another device.
upstream timed out (110: Connection timed out) while reading response header from upstream Probably because Nginx tries to get the data from puma again and again.The funny thing is that the error caused the timeout message even if I'm calling a different action in the controller, so, a single typo blocks all the app.
This caused upstream to sometimes fall in timeout and drop the connection, while nginx didn't understand why. Raising the upstream server value to match nginx' one solved the issue.
That's it and thousand of TIME_WAIT connections in backend are gone, there are only round about 150 now instead of 30-40k. Would you happen to know what response code is sent back to the client in case the upstream timed out error comes?
Same thing happened to me and I used 1 hour timeout for an internal app at work: With this, NGINX will wait for an hour (3600s) for its upstream to return something. Show activity on this post. You should always refrain from increasing the timeouts, I doubt your backend server response time is the issue here in any case.
OK, I didn't like reading manuals, but to answer my question:
nginx close upstream connection after request
solved it. So what was the problem: I've configured upstream to use keepalive
but Nginx doc suggests to set following options in proxy location:
proxy_http_version 1.1;
proxy_set_header Connection "";
That's it and thousand of TIME_WAIT
connections in backend are gone, there are only round about 150 now instead of 30-40k.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With