Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Hystrix circuitBreaker.sleepWindowInMilliseconds not working

I'm having a spring boot application which is calling iteratively a mockserver instance via a hystrix command, with a fallback method.

The mockserver is configured to allways respond with status code 500. When running without having circuitBreaker.sleepWindowInMilliseconds, everything works fine, the call is done to the mockserver and then the fallback method is invoked.

After configuring circuitBreaker.sleepWindowInMilliseconds value to 5 minutes or so, I would expect that during 5 minutes no calls are done to the mockserver all the calls being directed to the fallback method, but that's not the case.

It looks like the circuitBreaker.sleepWindowInMilliseconds configuration is ignored.

For instance if I reconfigure the mockservice to reply with status code 200 while the iteration is still running, it would imediately print the "mockservice response",without waiting 5 minutes.

in the spring boot main application class:

@RequestMapping("/iterate")
  public void iterate() {
    for (int i = 1; i<100; i++ ) {
      try {
        System.out.println(bookService.readingMockService());
        Thread.sleep(3000);
      } catch (Exception e) {
        System.out.println(e.getMessage());
      }
    }
  }

in the spring boot service :

@HystrixCommand(groupKey = "ReadingMockService", commandKey = "ReadingMockService", threadPoolKey = "ReadingMockService", fallbackMethod = "reliableMock", commandProperties = {
          @HystrixProperty(name ="circuitBreaker.sleepWindowInMilliseconds", value = "300000") })
  public String readingMockService() {
    URI uri = URI.create("http://localhost:1080/askmock");
    return this.restTemplate.getForObject(uri, String.class);
  }

also the mock server is running on the same machine, being configured like :

new MockServerClient("127.0.0.1", 1080).reset();
   new MockServerClient("127.0.0.1", 1080)
           .when(request("/askmock"))
           .respond(response()
                   .withStatusCode(500)
                   .withBody("mockservice response")
                   .applyDelay());
like image 529
DariusNica Avatar asked Oct 18 '22 08:10

DariusNica


2 Answers

Found the problem : This property (...circuitBreaker.sleepWindowInMilliseconds ) works together with another one (...circuitBreaker.requestVolumeThreshold ). If not specifically set this defaults to 20, meaning that first hystrix will try to connect 20 times the usual way and only afterwards the sleepWindowInMilliseconds will get activated and will go to fallback only.

Also the circuit break opens only if the percentage of failed calls exceeds circuitBreaker.errorThresholdPercentage and in the same time the total number of failed calls exceeds circuitBreaker.requestVolumeThreshold, all within a window of metrics.rollingStats.timeInMilliseconds

like image 89
DariusNica Avatar answered Oct 21 '22 02:10

DariusNica


From the docs: https://github.com/Netflix/Hystrix/wiki/configuration#circuitBreaker.sleepWindowInMilliseconds

and by looking at the source code:

https://github.com/Netflix/Hystrix/blob/master/hystrix-core/src/main/java/com/netflix/hystrix/HystrixCommandProperties.java

Using

@HystrixProperty(name="hystrix.command.ReadingMockService.circuitBreaker.sleepWindowInMilliseconds"

should work.

like image 26
Olga Khylkouskaya Avatar answered Oct 21 '22 02:10

Olga Khylkouskaya