nginx proxy: connect() to ip:80 failed (99: Cannot assign requested address)

Question

An nginx/1.0.12 running as a proxy on Debian 6.0.1 starts throwing the following error after running for a short time:

connect() to upstreamip:80 failed (99: Cannot assign requested address) 
while connecting to upstream, client: xxx.xxx.xxx.xxx, server: localhost, 
request: "GET / HTTP/1.1", upstream: "http://upstreamip:80/", 
host: "requesteddomain.com"

Not all requests produce this error, so I suspect that it has to do with the load of the server and some kind of limit it hit.

I have tried raising ulimit -n to 50k and worker_rlimit_nofile to 50k as well, but that does not seem to help. lsof -n shows a total of 1200 lines for nginx. Is there a system limit on outgoing connections that might prevent nginx from opening more connections to its upstream server?

Answer 1

Seems like I just found the solution to my own question: Allocating more outgoing ports via

echo "10240 65535" > /proc/sys/net/ipv4/ip_local_port_range

solved the problem.

Answer 2

Each TCP connection has to have a unique quadruple source_ip:source_port:dest_ip:dest_port

source_ip is hard to change, source_port is chosen from ip_local_port_range but can't be more than 16 bits. The other thing left to adjust is dest_ip and/or dest_port. So add some IP aliases for your upstream server:

upstream foo { server ip1:80; server ip2:80; server ip3:80; }

Where ip1, ip2 and ip3 are different IP addresses for the same server.

Or it might be easier to have your upstream listen on more ports.