Unable to install confluent-kafka: "fatal error: librdkafka/rdkafka.h: No such file or directory"

Question

I am using the confluent-kafka Python client in my project. I'm trying to create a Docker image with this client.

I am facing the following error:-

#11 8.015 [pipenv.exceptions.InstallError]:       In file included from /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/Admin.c:17:
#11 8.015 [pipenv.exceptions.InstallError]:       /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
#11 8.015 [pipenv.exceptions.InstallError]:          23 | #include <librdkafka/rdkafka.h>
#11 8.015 [pipenv.exceptions.InstallError]:             |          ^~~~~~~~~~~~~~~~~~~~~~
#11 8.015 [pipenv.exceptions.InstallError]:       compilation terminated.
#11 8.015 [pipenv.exceptions.InstallError]:       error: command '/usr/bin/gcc' failed with exit code 1
#11 8.016 [pipenv.exceptions.InstallError]:       [end of output]

Based on my search it is related to Apple M1 build for librdkafka.

Answer 1

If you are installing confluent-kafka inside Docker container on M1, this helped me.

RUN apt-get update && \
  apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && \
  cd /tmp && git clone https://github.com/edenhill/librdkafka.git && \
  cd librdkafka && git checkout tags/v1.9.0 && \
  ./configure && make && make install && \
  cd ../ && rm -rf librdkafka