Connection timeout using local kafka-connect cluster to connect on a remote database

Question

I'm trying to run a local kafka-connect cluster using docker-compose. I need to connect on a remote database and i'm also using a remote kafka and schema-registry. I have enabled access to these remotes resources from my machine.

To start the cluster, on my project folder in my Ubuntu WSL2 terminal, i'm running

docker build -t my-connect:1.0.0

docker-compose up

The application runs successfully, but when I try to create a new connector, returns error 500 with timeout.

My Dockerfile

FROM confluentinc/cp-kafka-connect-base:5.5.0

RUN cat /etc/confluent/docker/log4j.properties.template

ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"
ARG JDBC_DRIVER_DIR=/usr/share/java/kafka/

RUN   confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:5.5.0 \
   && confluent-hub install --no-prompt confluentinc/connect-transforms:1.3.2

ADD java/kafka-connect-jdbc /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib/
COPY java/kafka-connect-jdbc/ojdbc8.jar /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib/

ENTRYPOINT ["sh","-c","export CONNECT_REST_ADVERTISED_HOST_NAME=$(hostname -I);/etc/confluent/docker/run"] 

My docker-compose.yaml

services:
  connect:
    image: my-connect:1.0.0
    ports:
     - 8083:8083
    environment:
      - CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http=//schema-registry:8081
      - CONNECT_KEY_CONVERTER=io.confluent.connect.avro.AvroConverter
      - CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http=//schema-registry:8081
      - CONNECT_BOOTSTRAP_SERVERS=broker1.intranet:9092
      - CONNECT_GROUP_ID=kafka-connect
      - CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
      - CONNECT_VALUE_CONVERTER=io.confluent.connect.avro.AvroConverter
      - CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
      - CONNECT_OFFSET_STORAGE_TOPIC=kafka-connect.offset
      - CONNECT_CONFIG_STORAGE_TOPIC=kafka-connect.config
      - CONNECT_STATUS_STORAGE_TOPIC=kafka-connect.status
      - CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY=All
      - CONNECT_LOG4J_ROOT_LOGLEVEL=INFO
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - CONNECT_REST_ADVERTISED_HOST_NAME=localhost

My cluster it's up

~$ curl -X GET http://localhost:8083/
{"version":"5.5.0-ccs","commit":"606822a624024828","kafka_cluster_id":"OcXKHO7eT4m9NBHln6ACKg"}

Connector call

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d
{
    "name": "my-connector",
    "config":  
    { 
    "connector.class" : "io.debezium.connector.oracle.OracleConnector",
    "tasks.max": "1",
    "database.user": "user", 
    "database.password": "pass",    
    "database.dbname":"SID",
    "database.schema":"schema",
    "database.server.name": "dbname",   
    "schema.include.list": "schema",    
    "database.connection.adapter":"logminer",   
    "database.hostname":"databasehost",
    "database.port":"1521"
   }
}

Error

{"error_code": 500,"message": "IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}

## LOG
connect_1  | [2021-07-01 19:08:50,481] INFO Database Version: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
connect_1  | Version 19.4.0.0.0 (io.debezium.connector.oracle.OracleConnection)
connect_1  | [2021-07-01 19:08:50,628] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
connect_1  | [2021-07-01 19:08:50,643] INFO AbstractConfig values:
connect_1  |  (org.apache.kafka.common.config.AbstractConfig)
connect_1  | [2021-07-01 19:09:05,722] ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestClient)
connect_1  | java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout

Testing the connection to the database

$ telnet databasehostname 1521 Trying <ip>... Connected to databasehostname

Testing connection to kafka broker

$ telnet broker1.intranet 9092 Trying <ip>... Connected to broker1.intranet

Testing connection to remote schema-registry

$ telnet schema-registry.intranet 8081 Trying <ip>... Connected to schema-registry.intranet

What am I doing wrong? Do I need to configure something else to allow connection to this remote database?

Answer 1

You need to set correctly rest.advertised.host.name (or CONNECT_REST_ADVERTISED_HOST_NAME, if you’re using Docker). This is how a Connect worker communicates with other workers in the cluster.

For more details see Common mistakes made when configuring multiple Kafka Connect workers by Robin Moffatt.

In your case try to remove CONNECT_REST_ADVERTISED_HOST_NAME=localhost from compose file.