Spring cloud Zuul retry when instance is down and forward to other available instance

Question

using 'Camden.SR5' for spring-cloud-dependencies, with spring boot '1.5.2.RELEASE'.

In my current setup, I have

• eureka server
• config server (running on random ports)
• zuul gateway server
• and 2 instances of a service (running on random ports)

All these instances are successfully register with Eureka.

When all the services are running, The load balancing is done properly through zuul without any issues.

when an instance is killed, Zuul is still trying to fulfil the request using the same service which is down. However if waited till the eureka registry is fetched after shutting down the instance, requests are fulfilled with the other instances which are 'UP'.

    2017-03-07 19:57:41.409 DEBUG 26658 --- [nio-5555-exec-3] c.n.l.reactive.LoadBalancerCommand       : Got error org.apache.http.conn.HttpHostConnectException: Connect to 10.99.4.151:64381 [/10.99.4.151] failed: Connection refused when executed on server 10.99.4.151:64381
2017-03-07 19:57:41.420 DEBUG 26658 --- [nio-5555-exec-3] com.netflix.hystrix.AbstractCommand      : Error executing HystrixCommand.run(). Proceeding to fallback logic ...

com.netflix.client.ClientException: null
    at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:123) ~[ribbon-loadbalancer-2.2.0.jar:2.2.0]
    at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:81) ~[ribbon-loadbalancer-2.2.0.jar:2.2.0]
    at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:96) ~[spring-cloud-netflix-core-1.2.5.RELEASE.jar:1.2.5.RELEASE]
    at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:42) ~[spring-cloud-netflix-core-1.2.5.RELEASE.jar:1.2.5.RELEASE]

---

    at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.3.jar:4.5.3]
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.3.jar:4.5.3]
    ... 162 common frames omitted

2017-03-07 19:57:41.425 DEBUG 26658 --- [nio-5555-exec-3] com.netflix.hystrix.AbstractCommand      : No fallback for HystrixCommand. 

java.lang.UnsupportedOperationException: No fallback available.
    at com.netflix.hystrix.HystrixCommand.getFallback(HystrixCommand.java:292) [hystrix-core-1.5.6.jar:1.5.6]
    at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.getFallback(AbstractRibbonCommand.java:117) ~[spring-cloud-netflix-core-1.2.5.RELEASE.jar:1.2.5.RELEASE]
    at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.getFallback(AbstractRibbonCommand.java:42) ~[spring-cloud-netflix-core-1.2.5.RELEASE.jar:1.2.5.RELEASE]

---

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.11.jar:8.5.11]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]

2017-03-07 19:57:41.428  WARN 26658 --- [nio-5555-exec-3] o.s.c.n.z.filters.post.SendErrorFilter   : Error during filtering

com.netflix.zuul.exception.ZuulException: Forwarding error
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:170) ~[spring-cloud-netflix-core-1.2.5.RELEASE.jar:1.2.5.RELEASE]
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:145) ~[spring-cloud-netflix-core-1.2.5.RELEASE.jar:1.2.5.RELEASE]
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:88) ~[spring-cloud-netflix-core-1.2.5.RELEASE.jar:1.2.5.RELEASE]

---

Following are the zuul configuration used with @EnableZuulProxy and @EnableEurekaClient

    server:
  port: 5555

spring:
  application:
    name: gateway-server
  cloud:
    config:
      discovery:
        enabled: true
        service-id: CONFIGSERVER
      fail-fast: true
      retry:
        multiplier:  1.1
        initial-interval: 1000
        max-attempts: 6
        max-interval: 2000

hystrix:
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 100000
        timeout:
          enabled: false

ribbon:
  ReadTimeout: 5000
  ConnectTimeout: 3000
  maxAutoRetries: 1
  MaxAutoRetriesNextServer: 2
  OkToRetryOnAllOperations: true


logging:
  level:
    ROOT: DEBUG

zuul:
  routes:
    security-service:
      retryable: true

---

The 2 instances of service with are running with unique instance-ids

@EnableEurekaClient
@EnableHystrix
@SpringBootApplication
public class SecurityServer implements HealthIndicator{

    public static void main(String args[])
    {
        SpringApplication.run(SecurityServer.class,args);
    }

    @Override
    public Health health() {
        return Health.up().withDetail("STATUS", "SUCCESS").build();
    }
}

---

instanceId: ${spring.cloud.client.hostname}:${spring.application.name}:${spring.application.instance_id:${random.uuid}}

---

Can you help me with the zuul & instances configuration, so that request is automatically forwarded to the other available instances when an instance goes down.

Answer 1

upon searching out more and looking into spring-cloud-netflix issue tracker, There is a wonderful discussion between william-tran and ryanjbaxter on the best practices. Thanks to both of you.

https://github.com/spring-cloud/spring-cloud-netflix/issues/1290#issuecomment-242204614

https://github.com/spring-cloud/spring-cloud-netflix/issues/1295

In summary, Camden doesn't use the Ribbon HTTP Client(deprecated) so none of the ribbon.* properties will help you control the retry logic. Camden uses Apache HTTP client.

So the solution would be to use Ribbon HTTP Client in camden version using below configuration

ribbon.restclient.enabled=true

or

Move to Camden.BUILD-SNAPSHOT or Dalston.BUILD-SNAPSHOT for using spring-retry (https://github.com/spring-projects/spring-retry)