Starting a Kafka topics using Docker Compose with spotify/kafka?

Question

I am attempting to connect Kafka topics to my front-end Java Spring application. I am utilizing Docker Compose and have tried to connect using two different Kafka images.

With wurstmeister/kafka I have been able to get the Kafka topics up and running by this service in my docker.compose.yml file. But I have not been able to connect the created topics to my front-end Java Spring application.

kafka:
    image: wurstmeister/kafka:0.10.2.0
    ports:
      - "9092:9092"
    expose:
      - "9092"
      - "2181"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_CREATE_TOPICS: "test-topic1:1:1, test-topic2:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper

Secondly, with spotify/kafka, I am having difficulty actually creating the topics with Kafka. In the documentation, it is looking for the topics as an environment variable, but the following docker-compose.yml service is not creating a topic. I have also tried putting quotes around the test-topic but that did not work as well.

kafka:
    image: spotify/kafka
    ports:
      - "9092:9092"
      - "2181:2181"
    hostname: kafka
    expose:
      - "9092"
      - "2181"
    environment:
      TOPICS: test-topic

I do not know if this is necessary, but my entire docker-compose.yml file is as follows, take note that the zookeeper service is only required if you use wurstmeister/kafka.

docker-compose.yml

version: '2'
services:
  # zookeeper:
  #   image: wurstmeister/zookeeper
  #   ports:
  #     - "2181:2181"
  kafka:
    image: spotify/kafka
    ports:
      - "9092:9092"
      - "2181:2181"
    hostname: kafka
    expose:
      - "9092"
      - "2181"
    environment:
      TOPICS: test-topic
  redis:
    image: redis
    ports:
      - "6379"
    restart: always
  kafka-websocket-connector:
    build: ./kafka-websocket-connector
    image: andrewterra/kafka-websocket-connector
    ports:
      - "8077:8077"
    #   - "9092:9092"
    depends_on:
      - kafka
      - redis
    #   - zookeeper
    links:
      - kafka
      - redis

Answer 1

Rather late, but you could use something like the following to use a shell script to create your topic:

command: >
  bash -c
  "(sleep 15s &&
  /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh
  --create
  --zookeeper
  localhost:2181 --replication-factor 1 --partitions 1
  --topic my_topic &) && (supervisord -n)"

Answer 2

Container run command to create topic

Use the container command docker run --net=host --rm. In the following example, the zookeeper is running on port 22181, please use the respective topic name, port.

Create

docker run   --net=host   --rm   confluentinc/cp-kafka:4.0.0   kafka-topics --create --topic customer --partitions 1 --replication-factor 1 --if-not-exists --zookeeper  localhost:22181

Describe

docker run   --net=host   --rm   confluentinc/cp-kafka:4.0.0   kafka-topics --zookeeper localhost:22181 --topic customer --describe

List

docker run   --net=host   --rm   confluentinc/cp-kafka:4.0.0   kafka-topics --list --zookeeper  localhost:22181

Delete

docker run   --net=host   --rm   confluentinc/cp-kafka:4.0.0   kafka-topics  --delete --topic customer --zookeeper localhost:22181