How do I set up a Kafka service on gitlab-ci.yml?

Question

I am currently having trouble setting up a Kafka service on gitlab CI to run integration tests on, I am currently using the spotify/kafka docker image. Would really appreciate it if someone could help me out with this.

Answer 1

It works nicely with a container that hosts both Kafka and Zookeeper, and allows control of both the internal and the external 'advertised listeners'. The external should be set to the alias of the container, the internal to localhost (that is: the container's localhost).

The spotify/kafka-image only allows you to set the advertised host, not the full advertised listener string; since internal and external must be different, that's not going to work.

The krisgeus/docker-kafka (https://github.com/krisgeus/docker-kafka) does allow to set everything needed.

The .gitlab-ci.yml below allows me to connect to kafka:9092 from the ci-jobs:

variables:
  ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092,INTERNAL://localhost:9093'
  LISTENERS: 'PLAINTEXT://0.0.0.0:9092,INTERNAL://0.0.0.0:9093'
  SECURITY_PROTOCOL_MAP: 'PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT'
  INTER_BROKER: 'INTERNAL'
  KAFKA_CREATE_TOPICS: 'cc-event:36:1'

services:
  - name: krisgeus/docker-kafka
    alias: kafka

Answer 2

I wasted days of my life trying to get this to work with docker containers, including the one mentioned spotify/kafka.

Initially I tried using it as a service. However no matter what I did it would not work. In fact my tests could connect to the service fine and the service did start up. However it looked like some weird networking issue between zookeeper and kafka. My guess is that you need to set ADVERTISED_HOST to spotify__kafka which allows connection between your runner and the service. However this means zookeeper thinks that kafka is also at that hostname instead of localhost and you run into a seemingly impossible to fix network issue as services cannot communicate with other services in gitlab. However this is just a guess so please feel free to
correct me if I am wrong.

Next I tried using docker in docker with the intention of starting kafka from within my tests. It was all going well again and the container started, however I realized that I was soon back to the exact issue I had above, in that localhost is not localhost when communicating with dind in gitlab, instead you need to use the hostname docker. Which again meant that zookeeper could not connect to kafka because ADVERTISED_HOST had to change.

In the end I said screw it and installed kafka manually without docker in my gitlab-ci.yml script. The script looks like this

stages:
  - build

build:
  stage: build
  image: ubuntu:latest
  variables:
    KAFKA_HOST: localhost:9092
  script:
    - apt-get update
    - apt-get install -y wget nodejs npm default-jre-headless
    - wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/kafka/2.1.0/kafka_2.11-2.1.0.tgz
    - tar -xzf kafka_2.11-2.1.0.tgz
    - ls -ltra kafka_2.11-2.1.0/bin
    - nohup kafka_2.11-2.1.0/bin/zookeeper-server-start.sh kafka_2.11-2.1.0/config/zookeeper.properties > /dev/null 2>&1 &
    - sleep 2
    - nohup kafka_2.11-2.1.0/bin/kafka-server-start.sh kafka_2.11-2.1.0/config/server.properties > /dev/null 2>&1 &
    - sleep 2
    - npm install
    - npm test

This is accessible on localhost. This isn't elegant i know, but it gets the job done