Eureka peers not synchronized

Question

I'm prototyping a set of Spring Cloud + Netflix OSS applications and have run into trouble with Eureka. In our setup, we have a Spring Cloud Config Server + Eureka Server, and then 2 modules that utilize that server component for bootstrapping and service discovery.

The problem I run into is that if I spin up 2 instances of the Eureka Server and try to pair them (based on the Two Peer Aware Eureka Servers in the docs) they don't synchronize with each other. See configs below and/or the code on GitHub.

Essentially, Peer1 starts up and looks fine. Peer2 will startup and look fine, with both peers showing each other in the services. However, if the "UserService" module spins up and registers itself with Peer1, Peer2 will never see it. If we then spin up the "Web" module pointing to Peer2, it can never resolve the UserService. They basically act in isolation.

I've tried several combinations of setting the serviceUrl both on the server and the instance of the Eureka servers but to no avail. Am I just configuring things wrong?

Peer 1 / default config:

server:
  port: 8888

eureka:
  dashboard:
    path: /dashboard
  instance:
    hostname: peer1
    leaseRenewalIntervalInSeconds: 3
  client:
      serviceUrl:
          defaultZone: ${eureka.server.serviceUrl:http://localhost:${server.port}/eureka/}
  server:
    serviceUrl:
        defaultZone: http://localhost:${server.port}/eureka/
        peer2: http://peer2/eureka/
    waitTimeInMsWhenSyncEmpty: 0


spring:
  application:
    name: demo-config-service
  profiles:
    active: native
  # required for Spring Cloud Bus
  rabbitmq:
    host: ${DOCKER_IP:192.168.59.103}
    port: 5672
    username: guest
    password: guest
    virtualHost: /
  cloud:
    config:
      server:
        prefix: /configs
        native:
          searchLocations: /Users/dave/workspace/oss/distributed-spring/modules/config-server/src/main/resources/testConfigs
#        git :
#          uri: https://github.com/joshlong/microservices-lab-configuration

Peer 2 config:

server:
  port: 8889

eureka:
  dashboard:
    path: /dashboard
  instance:
    hostname: peer2
    leaseRenewalIntervalInSeconds: 3
  client:
      serviceUrl:
          defaultZone: ${eureka.server.serviceUrl:http://localhost:${server.port}/eureka/}
  server:
    serviceUrl:
        defaultZone: http://localhost:8888/eureka/
        peer1: http://peer1/eureka/
    waitTimeInMsWhenSyncEmpty: 0


spring:
  application:
    name: demo-config-service
  profiles:
    active: native
  # required for Spring Cloud Bus
  rabbitmq:
    host: ${DOCKER_IP:192.168.59.103}
    port: 5672
    username: guest
    password: guest
    virtualHost: /
  cloud:
    config:
      server:
        prefix: /configs
        native:
          searchLocations: /Users/dave/workspace/oss/distributed-spring/modules/config-server/src/main/resources/testConfigs
#        git :
#          uri: https://github.com/joshlong/microservices-lab-configuration

Answer 1

There were a few problems. The defaultZone needs to be in the client section as noted in the docs. The defaultZone url needs the port.

/etc/hosts

127.0.0.1       peer1
127.0.0.1       peer2

Peer 1 Config (Partial)

eureka:
  instance:
    hostname: peer1
    leaseRenewalIntervalInSeconds: 3
  client:
    serviceUrl:
      defaultZone: http://peer2:8889/eureka/

Peer 2 Config (Partial)

eureka:
  dashboard:
    path: /dashboard
  instance:
    hostname: peer2
    leaseRenewalIntervalInSeconds: 3
  client:
    serviceUrl:
      defaultZone: http://peer1:8888/eureka/
  server:
    waitTimeInMsWhenSyncEmpty: 0

User service config (Partial) Config port was wrong.

spring:
  application:
    name: user-service
  cloud:
    config:
      uri: http://localhost:8888/configs

You can see user-service replicated to both peer1 and peer2. I can post a PR to your code if you want.

Peer 1

Peer 2

Answer 2

@spencergibb's didn't mention why this hack-ish workaround is required. There is a gotcha with running more than one Eureka server on the same host. Netflix code (com.netflix.eureka.cluster.PeerEurekaNodes.isThisMyUrl) filters out the peer URLs that are on the same host. This may have been done to prevent the server registering as its own peer (I’m guessing here) but because they don’t check for the port, peer awareness doesn’t work unless the Eureka hostnames in the eureka.client.serviceUrl.defaultZone are different. The hacky workaround for this is to define unique hostnames and then map them to 127.0.0.1 in the /etc/hosts file (or its Windows equivalent).

I've created a blog post with the details of Eureka here, that fills in some missing detail from Spring doc or Netflix blog. It is the result of several days of debugging and digging through source code.