Use Confluent Hub without Confluent Platform installation

Question

I am using these installation instructions for getting confluent hub client https://docs.confluent.io/current/connect/managing/confluent-hub/client.html

But, when I get to the line to install the kafka-connect-elasticsearch connector

confluent-hub install confluentinc/kafka-connect-elasticsearch:latest

I continue to get this error message:

Unable to detect Confluent Platform installation. Specify --component-dir and --worker-configs explicitly.

Error: Invalid options or arguments

I have ElasticSearch and Kafka all installed via Homebrew on my Mac

Answer 1

Specify --component-dir and --worker-configs explicitly

You'll have to pass both options to include the Homebrew location of Kafka Connect's plugin.path setting (where Confluent Hub would install to) and your connect-distributed.properties (so that the plugin path can be setup if it isn't already)

e.g. (on a Mac)

confluent-hub install <name> \
   --component-dir /usr/local/Cellar/kafka/plugins \
   --worker-configs /usr/local/etc/kafka/connect-distributed.properties

This command should download and extract the connector zip into /usr/local/Cellar/kafka/plugins and modify the connect-distributed.properties file to include that path in its plugin.path property

Here is an example using a Kafka Connect docker image that I've built

Answer 2

MacOS

In my case I have installed Kafka and Confluent Hub with brew and the installation path for Kafka is /usr/local/Cellar/kafka. And created a directory to install the connector.

Move to /usr/local/Caskroom/confluent-hub-client/6.2.1/bin, it is the directory confluent-hub is located.

Then I've executed

confluent-hub install confluentinc/kafka-connect-azure-blob-storage:1.6.2 \
  --component-dir /Users/Rustam_Zokirov/BigData/connectors \
  --worker-configs /usr/local/Cellar/kafka/3.0.0/libexec/config/connect-distributed.properties