Eureka's self-preservation mode never recovers

Question

I'm currently facing an issue where Eureka never clears out service instances that have become stale because a VM went down unexpectedly. Understandably, Eureka's self-preservation mode kicked in because there was a large drop (below the threshold) in service renewals/heartbeat requests. However, 15+ hours later the dead instances are still registered in Eureka. This is a major problem as service requests continue to be directed to the dead instances only to return errors.

My hope was that the threshold is continuously adjusted and after some period of time, Eureka's threshold would be at a new norm level and self-preservation mode would be reset. We are using Eureka in mirrored setup and our configurations are not very complex.

Our setup:

Eureka via spring-boot-starter-parent 1.2.5.RELEASE

eureka:
  dashboard:
    path: services
    enabled: false
  instance:
    hostname: localhost
    leaseRenewalIntervalInSeconds: 3
    metadataMap:
      managementPath: /admin
      instanceId: discoveryPrimary
  client:
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
  server:
    waitTimeInMsWhenSyncEmpty: 0

Is it possible to adjust Eureka configurations to reset the self-preservation mode (where it stops clearing instances) and allow it to clear service registries if the services are dead for 5+ minutes?

Answer 1

If you are having only a few instances of your services, everytime any of them fail, the self preservation will kick in because on default the renewalPercentThreshold is 0.85.

So if only 84% of your instances renewed their lease eureka "turns on" self preservation.

This means if you have 3 instances and one fails, only 66% percent of them renewed their licences so none will get deregistered. You can tune renewalPercentThreshold in the server properties to suite your deployment.

eureka:
  server:
    renewalPercentThreshold: 0.49

With this, if you have 2 instances and 1 fails you are still good.