Experiencing random timeouts for nginx proxy pass

Question

I have been battling this issue for some days now. I found a temporary solution but just can't wrap my head around what exactly is happening.

So what happens is that one request is handled immediately. And if I send the same request right after it hangs on 'waiting' for 60 seconds. If I cancel the request and send a new one it is handled correctly again. If I send a request after this one it hangs again. This cycle repeat. It sounds like a load-balancing issue but I didn't set it up. Does nginx have some sort of default load balancing for connection to the upstream server?

The error received is upstream timed out (110: Connection timed out).

I found out that changing these proxy parameters, it only hangs for 3 seconds and every subsequent request now handles fine (after the waited one). Because of a working keep-alive connection I suppose.

proxy_connect_timeout 3s;

It looks like setting up a connection to the upstream is timing out and then after the timeout it tries again and succeeds. Also in the "(cancelled)request - ok request - (cancelled)request" cycle described above there is no keep-alive being setup. Only if I wait for the request to complete. Which takes 60 seconds without the above settings and is unacceptable.

It happens for both domains..

NGINX conf:

worker_processes 1;

events
{
    worker_connections  1024;
}

http 
{
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    gzip on;

    # Timeouts
    client_body_timeout   12;
    client_header_timeout 12;
    send_timeout          10;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    include /etc/nginx/conf.d/*.conf;

    server
    {
        server_name  domain.com www.domain.com;
        root         /usr/share/nginx/html;
        index        index.html index.htm;

        location /api/
        {
           proxy_redirect off;
           proxy_pass http://localhost:3001/;
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection keep-alive;
           proxy_set_header Host $host;
           proxy_cache_bypass $http_upgrade;

           #TEMP fix
           proxy_connect_timeout 3s;
       }
}

DOMAIN2 conf:

server {
    server_name domain2.com www.domain2.com;

    location /api/
    {
        proxy_redirect     off;
        proxy_pass         http://localhost:5000/;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection keep-alive;
        proxy_set_header   Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-Proto $scheme;

        #TEMP fix
        proxy_connect_timeout 3s;
    }
}

Answer 1

I found the answer. However, I still don't fully understand why and how. I suspect setting up the keep-alive wasn't working as it should. I read to the documentation and found the answer there: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

For both the configuration files I added a 'upstream' block.

i.e.

DOMAIN2.CONF:

upstream backend
{
    server 127.0.0.1:5000;
    keepalive 16;
}

location /api/
{
    proxy_redirect     off;
    proxy_pass         http://backend/;
    proxy_http_version 1.1;
    proxy_set_header   Connection "";
    ...
    # REMOVED THE TEMP FIX
}

Make sure to:

• Clear the Connection header
• Use 127.0.0.1 instead of localhost in upstream block
• Set http version to 1.1