Using a connector with Helm-installed Kafka/Confluent

Question

I have installed Kafka on a local Minikube by using the Helm charts https://github.com/confluentinc/cp-helm-charts following these instructions https://docs.confluent.io/current/installation/installing_cp/cp-helm-charts/docs/index.html like so:

helm install -f kafka_config.yaml confluentinc/cp-helm-charts --name kafka-home-delivery --namespace cust360

The kafka_config.yaml is almost identical to the default yaml, with the one exception being that I scaled it down to 1 server/broker instead of 3 (just because I'm trying to conserve resources on my local minikube; hopefully that's not relevant to my problem).

Also running on Minikube is a MySQL instance. Here's the output of kubectl get pods --namespace myNamespace:

I want to connect MySQL and Kafka, using one of the connectors (like Debezium MySQL CDC, for instance). In the instructions, it says:

Install your connector

Use the Confluent Hub client to install this connector with:

confluent-hub install debezium/debezium-connector-mysql:0.9.2

Sounds good, except 1) I don't know which pod to run this command on, 2) None of the pods seem to have a confluent-hub command available.

Questions:

1. Does confluent-hub not come installed via those Helm charts?
2. Do I have to install confluent-hub myself?
3. If so, which pod do I have to install it on?

Answer 1

Ideally this should be configurable as part of the helm script, but unfortunately it is not as of now. One way to work around this is to build a new Docker from Confluent's Kafka Connect Docker image. Download the connector manually and extract the contents into a folder. Copy the contents of this to a path in the container. Something like below.

Contents of Dockerfile

FROM confluentinc/cp-kafka-connect:5.2.1
COPY <connector-directory> /usr/share/java

/usr/share/java is the default location where Kafka Connect looks for plugins. You could also use different location and provide the new location (plugin.path) during your helm installation.

Build this image and host it somewhere accessible. You will also have to provide/override the image and tag details during the helm installation.

Here is the path to the values.yaml file. You can find the image and plugin.path values here.

Answer 2

Just an add-on to Jegan's comment above: https://stackoverflow.com/a/56049585/6002912

You can choose to do the Dockerfile below. Recommended.

FROM confluentinc/cp-server-connect-operator:5.4.0.0

RUN confluent-hub install --no-prompt debezium/debezium-connector-postgresql:1.0.0

Or you can use a Docker's multi-stage build instead.

FROM confluentinc/cp-server-connect-operator:5.4.0.0

COPY --from=debezium/connect:1.0 \
    /kafka/connect/debezium-connector-postgres/ \
    /usr/share/confluent-hub-components/debezium-connector-postgres/

This will help you to save time on getting the right jar files for your plugins like debezium-connector-postgres.

From Confluent documentation: https://docs.confluent.io/current/connect/managing/extending.html#create-a-docker-image-containing-c-hub-connectors

Answer 3

The Kafka Connect pod should already have the confluent-hub installed. It is that pod you should run the commands on.