Spring Config server not reachable with docker-compose until client is restarted

Question

I have a 'discovery first' setup with Eureka, Config Server, and my client.

The issue is that these 3 services start in order, but the client-server seems to register too early, and can never find config-server. I've tried a third-party library that allows a wait until config-server:8888 is available, but that doesn't always seem to work either. It's similar to a race condition.

The workaround is that if I docker restart the client-server after everything is up, it registers and finds config-server just fine.

First run of docker-compose:

Fetching config from server at : http://localhost:8888
Connect Timeout Exception on Url - http://localhost:8888. Will be trying the next url if available

When I docker restart the client:

Fetching config from server at : http://a80b001d04a7:8888/
Located environment: name=client-server, profiles=[default], label=null, version=053c8e1b14dc0281d5af0349c9b2cf012c1a346f, state=null

Not sure if my JAVA_OPTS properties aren't being set fast enough from my docker-compose.yml, or there is some networking race condition, or what. I've been going back and forth on this for too long.

My configuration is below:

Here's my docker-compose.yml:

version: '3'
services:
  eureka:
    image: eureka-server:latest
    environment:
    - "JAVA_OPTS=-DEUREKA_SERVER=http://eureka:8761/eureka"
    ports:
      - 8761:8761
  config:
    image: config-server:latest
    environment:
      - "JAVA_OPTS=-DEUREKA_SERVER=http://eureka:8761/eureka"
    depends_on:
      - eureka
    ports:
      - 8888:8888
  client:
    image: client-server:latest
    environment:
      JAVA_OPTS: -DEUREKA_SERVER=http://eureka:8761/eureka
    depends_on:
      - config
    ports:
      - 9000:9000

Here's the eureka-server application.yml:

server:
  port: 8761

spring:
  application:
    name: eureka-server

eureka:
  client:
    registerWithEureka: false
    fetchRegistry: false
    service-url:
      defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}

Here's the config-server bootstrap.yml:

server:
  port: 8888

eureka:
  client:
    serviceUrl:
      defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}

spring:
  application:
    name: config-server

Here's the client-server bootstrap.yml:

spring:
  application:
    name: client-server
  cloud:
    config:
      discovery:
        enabled: true
        serviceId: config-server
      fast-fail: true
    retry:
      max-attempts: 10000
      max-interval: 1000

eureka:
  instance:
    hostname: client-server
  client:
    registerWithEureka: true
    fetchRegistry: true
    serviceUrl:
      defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}

Edit:

Using the docker-compose wait library (https://github.com/ufoscout/docker-compose-wait), I can have the client-server wait for eureka and config to be available, then wait 90 seconds (Eureka documentation suggests that registration could take up to 90 seconds), and it seems to work consistently.

Is this an acceptable solution? Feels like a bit of a hack.

Answer 1

Being purist the answer to your question is NO, it is not an acceptable solution, because as it is stated here, Docker removed healthcheck from v3 on for some reason:

Docker have made a conscious decision not to support features that wait for containers to be in a "ready" state. They argue that applications depending on other systems should be resilient to failure.

In the same link, it is described why:

The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.

To handle this, your application should attempt to re-establish a connection to the database after a failure. If the application retries the connection, it should eventually be able to connect to the database.

Basically then, there are three options:

1. Use v2.1 with healhcheck. See an example here
2. Use v3 and a tool like wait-for-it or dockerize as @ortomala-lokni already perfectly explained
3. Make your application resilient to config-server failure and able config-client to retry the connection on startup

The recommended and acceptable solution is 3). You can use Spring Retry as it is mentioned here. Find below the bootstrap.yml configuration:

spring:
  application:
    name: config-client
  profiles:
     active: dev
  cloud:
    config:
     discovery:
       enabled: true
       service-id: config-server
     fail-fast: true
     retry:
       initial-interval: 1500
       multiplier: 1.5
       max-attempts: 10000
       max-interval: 1000

eureka:
  instance:
    hostname: config-client
  client:
    registerWithEureka: true
    fetchRegistry: true
    serviceUrl:
      defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}

BTW I found an error in your spring configuration. It is fail-fast and not fast-fail.

Remember to include the following dependencies (or similar if you are using gradle):

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>

You can find a very well configuration (and explanation) here taking also into account resiliency during the registering process in the Eureka Server.

When having a microservices environment we must think of the resiliency of our environment when platform services like config-service, discovery-service are not available for a short period of time.

But I am not a purist at all and I would not have removed some functionality people is using (it is a question of freedom). So, an alternative solution is:

If it is working for you, then go ahead

Because I do not really understand why Docker suppressed the fantastic healthcheck command from v3.

Answer 2

The best solution is probably, as Carlos Cavero said, to make your application resilient to config-server failure. But you can also solve the problem by using the wait-for script from Eficode on Github.

Copy the script into your container and in your docker-compose.yml use:

client:
    image: client-server:latest
    environment:
      JAVA_OPTS: -DEUREKA_SERVER=http://eureka:8761/eureka
    depends_on:
      - config
    ports:
      - 9000:9000
    command: wait-for $CONFIGSERVER_SERVICE_NAME:$CONFIGSERVER_PORT -- java $JVM_OPTIONS -jar client.war $SPRING_OPTIONS

The environment variables for CONFIGSERVER_SERVICE_NAME and CONFIGSERVER_PORT can be defined in your Docker Compose environment file.

If you need to wait for multiple services, you can merge this pull request and list all needed services in the command line parameters such as:

command: wait-for $SERVICE1_NAME $SERVICE1_PORT $SERVICE2_NAME $SERVICE2_PORT -- java $JVM_OPTIONS -jar client.war $SPRING_OPTIONS