Docker Compose with Zookeeper, Kafka, Redis, and Java Spring Boot

Question

I'm having difficulty linking these containers together using Docker Compose, let me preface by saying that I am currently running on a Mac as well.

The application is currently working without the use of Docker Compose, in that if I run these all individually (not with Docker) the application works as intended.

As intended means that the application is reading from Redis as well as pulling data that comes across certain Kafka topics and displaying them on the front-end.

On to what I believe are the necessary files.

docker-compose.yml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:0.10.2.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_CREATE_TOPICS: "flight-events:1:1,reported-flight-time-events:1:1,pax-flight-events:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper
  redis:
    image: redis
    ports:
      - "6379"
    restart: always
  kafka-websocket-connector:
    build: ./kafka-websocket-connector
    image: <user>/kafka-websocket-connector
    ports:
      - "8077:8077"
    depends_on:
      - kafka
      - redis
      - zookeeper
    links:
      - "kafka"
      - "redis"

The Dockerfile that is being referenced under kafka-websocket-connector is as follows:

Dockerfile

FROM frolvlad/alpine-oraclejdk8:slim
VOLUME /tmp
ADD target/kafka-websocket-connector-0.0.1-SNAPSHOT.jar app.jar
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]

If I attempt to run the command docker-compose up --build I receive the following error:

ERROR: Service 'kafka-websocket-connector' failed to build: ADD failed:      stat /var/lib/docker/tmp/docker-builder838069739/target/kafka-websocket-connector-0.0.1-SNAPSHOT.jar: no such file or directory

I don't know if this necessarily relates to the connecting these components, but my build.gradle for the Spring application is as follows:

build.gradle

buildscript {
    ext {
        springBootVersion = '1.5.4.RELEASE'
    }
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")

        classpath('se.transmode.gradle:gradle-docker:1.2')
    }
}

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'org.springframework.boot'

apply plugin: 'docker'

group = '<user>'

jar {
    baseName = 'kafka-websocket-connector'
    version = '0.0.1-SNAPSHOT'
    sourceCompatibility = 1.8
}

task buildDocker(type: Docker, dependsOn: build) {
  applicationName = jar.baseName
  dockerfile = file('Dockerfile')
  doFirst {
    copy {
      from jar
      into "${stageDir}/target"
    }
  }
}



repositories {
    mavenCentral()
}


ext {
    springCloudVersion = 'Dalston.SR1'
}

dependencies {

    compile('org.springframework.boot:spring-boot-starter-data-redis')
    compile('org.springframework.kafka:spring-kafka')
    compile('org.springframework.boot:spring-boot-starter-web')
    compile('org.springframework.boot:spring-boot-starter-websocket')
    compile("org.webjars:webjars-locator")
    compile("org.webjars:sockjs-client:1.0.2")
    compile("org.webjars:stomp-websocket:2.3.3")
    compile("org.webjars:bootstrap:3.3.7")
    compile("org.webjars:jquery:3.1.0")
    compile group: 'org.webjars', name: 'd3js', version: '4.2.1'
    testCompile('org.springframework.boot:spring-boot-starter-test')
    testCompile('org.springframework.restdocs:spring-restdocs-mockmvc')

}

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
    }
}

bootRepackage {
    executable = true
    enabled = true
}

So to restate the problem, I am having trouble connecting the following components using Docker Compose: Zookeeper, Kafka, Redis, and a Spring Boot application.

Answer 1

The KAFKA_ADVERTISED_HOST_NAME: localhost environment variable in your kafka service is likely the cause of the problem. It needs to be reachable by other containers. Try change it to KAFKA_ADVERTISED_HOST_NAME: kafka.

All the containers of the services defined in your docker-compose.yml are added to a user-defined network, named after your compose project. The containers in this network are reachable and discoverable by other each, using hostnames that are identical to the container name. You can find out more information on Docker Compose networking in their documentation.