Gateway timeout with Spring cloud gateway and Nginx as reverse proxy

Question

I created an API gateway for my application and it will act as a front controller for other microservices. In my production setup I use Nginx as a reverse proxy for my gateway

The API gateway is running on port 8080

Nginx configured as below:

gateway-api.conf:

server {
    listen 80;
    server_name api.example.com;
    location / {
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://localhost:30010/;
        keepalive_timeout 500s;
    }
    keepalive_timeout 500s;
    access_log /var/log/nginx/api.log;  
    error_log /var/log/nginx/api_error.log;
}

Timeout setting in nginx.conf:

proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;

Spring cloud gateway gradle file:

compile('org.springframework.cloud:spring-cloud-starter-gateway')
 compile('org.springframework.cloud:spring-cloud-starter-openfeign')
 compile("org.springframework.boot:spring-boot-starter-actuator")
 compile('org.springframework.boot:spring-boot-starter-security')

springBootVersion=2.0.3.RELEASE
springDMPVersion=1.0.4.RELEASE
springPlatformBomVersion=Cairo-SR2
springCloudVersion=Finchley.RELEASE

Gateway application:

@SpringBootApplication
@ComponentScan(basePackages = {"com.example"})
@EntityScan(basePackages = {"com.example"})
@EnableFeignClients(basePackages = "com.example")
public class GatewayApplication {

    public static void main(String[] args) {
        SpringApplication.run(GatewayApplication.class, args);
    }
}

Problem statement:

In one of my microservices, one REST API takes more than 3 minutes to complete. If I call this API via nginx(api.example.com), it fails exactly after 1 minute and gives HTTP status 504.

curl:

curl --request GET \
  --url http://api.example.com/hellomicroservice/api/take/moretime

error:

504 Timeout while reading the response from Server

There are no error logs in nginx or the API gateway.

Access log from nginx:

203.129.213.102 - - [01/Apr/2019:08:14:33 +0000] "GET hellomicroservice/api/take/moretime HTTP/1.1" 499 0 "-" "PostmanRuntime/7.3.0"

But when I make a call of the same API directly to the gateway (on gateway port 8080), the request is processed successfully.

curl with gateway port:

curl --request GET \
  --url http://api.example.com:8080/hellomicroservice/api/take/moretime

---

Edit:

If I apply the Nginx timeout settings as less than 60 Seconds (e.g. 30 seconds), the request gets timed out in a specified time interval. But if I set the Nginx timeout to be more than 60 seconds, let's 300 Seconds, the request gets timed out after 60 seconds.

Answer 1

It seems the request time-outs are not the problem for you. Its the connection timeout. I think we need to look on the header of

Connection

AFAIK, the Connection header defines, the connection should be persistent or who has the authority to maintain/close it. If the connection was keep-alive , then the connection will be persistent. For the keep-alive connections, the client occasionaly sends a TCP ping to ensure the server is still alive and holding the connection. As per the curl this time by default is every 60 seconds.

Now the nginx has to be configured to accept connections and keep it alive for a while using the keepalive_timeout directive. If this is not there, then nginx will not keep the connections alive.

This should be the reason why nginx says 499 in the logs. HTTP499 is a cutom error in nginx which says the client closed the connection. In your case the curl closed it. Why curlclosed it? because nginx did not respond to the TCP ping of 60 seconds as the keep-alive is not enabled.

Adding the keepalive_timeout to ~500 or a higher value than the application timeout should solve your problem.

Now, why it worked with tomcat directly? I think spring enables the alive timeout to be infinite or a very higher value. Normally in tomcat also its 60 seconds.

I hope this solves your problem.