KAFKA: Failed to update metadata after 60000 ms

Question

I am new to KAFKA and I know that this question has been asked multiple times on stack overflow but none of the solutions worked for me so here I am trying my luck with asking the same question again. I have downloaded and installed KFKA on Centos7 VM. The VM is on my laptop. When I run the KAFKA producer and consumer from the command line, it works fine. Next step, I wanted to create a Java Producer but it always timeout with the following exception.

    Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
    at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1186)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:880)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:690)
    at com.soft.teradata.KafkaProducerExample.runProducer(KafkaProducerExample.java:40)
    at com.soft.teradata.KafkaProducerExample.main(KafkaProducerExample.java:55)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

The Java code for the producer is:

package com.soft;
import java.util.Properties;
import java.util.concurrent.Future;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
public class SimpleProducer {
    public static void main(String[] args) throws Exception {
        try {
            String topicName = "Hello-Kafka";
            Properties props = new Properties();
            props.put("bootstrap.servers", "192.168.xxx.xxx:9092");
            props.put("acks", "all");
            props.put("retries", 1);
            props.put("batch.size", 16384);
            props.put("linger.ms", 1);
            props.put("buffer.memory", 33554432);
            props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
            props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
            Producer<String, String> producer = new KafkaProducer<String, String>(props);
            Future<RecordMetadata> f=producer.send(new ProducerRecord<String, String>(topicName, "Eclipse"));
            System.out.println("Message sent successfully");
            producer.close();
        } catch (Exception e) {
            System.out.println(e.getMessage());
        } 
            System.out.println("Successful");

    }
} 

For bootstrap.server, i have even tried the following:

props.put("bootstrap.servers", "PLAINTEXT://192.168.xxx.xxx:9092");

Please note that I am executing the java code from Eclipse on my laptop and KAFKA is installed on a CENTOS7 VM on my laptop. 192.168.xxx.xxx is the IP address of the CENTOS7 VM. I have noticed that 192.168.xxx.xxx:9092 (telnet 192.168.xxx.xxx 9092) is inaccessible from my laptop. I added the port to firewall but still no success.

firewall-cmd --zone=public --add-port=9092/tcp --permanent
        firewall-cmd --reload
        firewall-cmd --list-ports

The version of KAFKA is 2.12-2.0.0 and I have added the following jars to my Eclipse Classpath:

• kafka-clients-2.0.0.jar
• lz4-java-1.4.1.jar
• slf4j-api-1.7.25.jar
• snappy-java-1.1.7.1.jar

Thanks a lot for the help in advance :)

Regadrs, DIRSHAH.

Answer 1

This error can indicate that the topic doesn't exist, so you may want to double check your topicName = "Hello-Kafka".

Though this isn't exactly a deep answer, it seems to be a common problem, see also https://github.com/dpkp/kafka-python/issues/607

Answer 2

I faced the same issue. You need to advertise the hostname/ip of Kafka broker to be reachable from Kafka Producer pc.

 kafka-server-start.sh config/server.properties --override  advertised.listeners=PLAINTEXT://<accessible-hostname>:9092