Building Go Application using confluent-kafka-go on Linux

Question

I am trying to create a docker image with my go application. The application (which was developed on MacOS) depends on confluent-kafka-go which in turn depends on librdkafka-dev which I install in the Docker image like so:

FROM golang:1.1
RUN apt-get update
RUN apt-get -y install librdkafka-dev

VOLUME /workspace
WORKDIR /workspace/src/my/app/folder
ENTRYPOINT ["/bin/sh", "-c"]

I am getting the following error:

my/app/folder/vendor/github.com/confluentinc/confluent-kafka-go/kafka ../folder/vendor/github.com/confluentinc/confluent-kafka-go/kafka/00version.go:44:2: error: #error "confluent-kafka-go requires librdkafka v0.11.5 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html"

As far as I understand the latest version is installed. How can I fix it?

Answer 1

I had a similar issue a few weeks ago. IIRC confluent-kafka-go requires a recent version of librdkafka-dev, which simply was not yet released to alpine or others. I was able to find it for ubuntu though, so my solution (which was more involved than I hoped for, but it worked), was to start from clean ubuntu, install librdkafka-dev, install Go version that I want and compile inside docker.

Here's how it looks:

FROM ubuntu

# Install the C lib for kafka
RUN apt-get update
RUN apt-get install -y --no-install-recommends apt-utils wget gnupg software-properties-common
RUN apt-get install -y apt-transport-https ca-certificates
RUN wget -qO - https://packages.confluent.io/deb/5.1/archive.key | apt-key add -
RUN add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/5.1 stable main"
RUN apt-get update
RUN apt-get install -y librdkafka-dev

# Install Go
RUN add-apt-repository ppa:longsleep/golang-backports
RUN apt-get update
RUN apt-get install -y golang-1.11-go

# build the library
WORKDIR /go/src/gitlab.appsflyer.com/rantav/kafka-mirror-tester
COPY *.go ./
COPY // the rest of your go files. You may copy recursive if you want
COPY vendor vendor

RUN GOPATH=/go GOOS=linux /usr/lib/go-1.11/bin/go build -a -o main .

EXPOSE 8000

ENTRYPOINT ["./main"]