Getting 504 timeout error in a GRPC call under Nginx web server

Question

I am using a unary RPC method. I have a server that I have configured under the Nginx web server. Following is the Nginx configuration:

server {
        listen 80 http2;
        server_name test.grpc.tester.local;


        access_log /var/log/nginx/test.grpc.tester.local.access.log;
        error_log /var/log/nginx/test.grpc.tester.local.error.log;

        location / {
            grpc_pass grpc://localhost:8001;
        }
}

I have a client that is also available in the same host machine. I have called this unary method in couple of ways and I have following observation:

1. If I call the server RPC method using the Nginx proxy (test.grpc.tester.local:80) and if it takes more than 1 minutes then I get errors:

   • in error logs of Nginx: upstream timed out (110: Connection timed out) while reading response header from upstream

   • in the client log: Error: 1 CANCELLED: Received http2 header with status: 504.

2. If I don't use the Nginx and directly access using localhost:8001, it never gives the error (even server takes more than 1 minute) and gives success status.

My query over here is that how can I increase the request time, so I won't get the timeout error. I have tried these following configs but no luck:

    proxy_connect_timeout  600s;
    proxy_send_timeout  600s;
    proxy_read_timeout  600s;
    fastcgi_send_timeout 600s;
    fastcgi_read_timeout 600s;

Can anyone guide me on how can I resolve this issue?

Thank you.

Answer 1

I ran into the same issue and it appears to be fixed with the following configuration;

location /grpweburl
{
    proxy_pass http://localhost:5000/grpweburl;
    proxy_request_buffering off;
    proxy_buffering off;
    proxy_connect_timeout  600s;
    proxy_send_timeout  600s;
    proxy_read_timeout  600s;
}

Do note - I am using grpc-web. Not just grpc. But I suspect adding the buffering settings will fix your issue as it did for me.

I am uncertain why exactly this combination works, but it did for me. The buffering part was to make sure data-updates are sent immediately to the client.

In combination with the timeout settings I am now only getting a http-504 out of nginx every 10 minutes if the grpc-service gives no results. As expected.

In order not to mess with normal operations I only set these config settings for the grpc-web url(s).