Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

kafka cluster configuration

I'm new on Kafka. I have question about kafka configuration.

I want to using seperate server like below,

server1: kafka producer server2: kafka broker, kafka consumer, zookeeper

But, I can't send message to broker. And I got this error messages.

on console-producer(server1), console stdout error message `

[2016-05-24 16:41:11,823] ERROR Error when sending message to topic twitter with key: null, value: 3 bytes with error: Failed to update metadata after 60000 ms.(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

`

on kafka producer(server2), console stdout error message `

[2016-05-25 10:20:01,588] DEBUG Connection with /192.168.50.142 disconnected (org.apache.kafka.common.network.Selector)
java.io.EOFException
    at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:160)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:141)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
    at kafka.network.Processor.run(SocketServer.scala:413)
    at java.lang.Thread.run(Thread.java:745)

`

running commands are like below

server1 on kafka dir, `

./bin/zookeeper-server-start.sh config/zookeeper.properties
./bin/kafka-server-start.sh config/server.properties
./bin/kafka-console-consumer.sh --zookeeper 192.168.50.142:2181 --from-beginning --topic twitter
./bin/kafka-topics.sh --create --zookeeper 192.168.50.142:2181 --replication-factor 1 --partitions 1 --topic twitter

`

and server2 on kafka dir, `

./bin/kafka-console-producer.sh --broker-list 192.168.50.142:9092 --topic twitter

`

And my configuration are,

server1(IP: 192.168.50.155):

kafka/config/producer.properties `

metadata.broker.list=192.168.50.142:9092
producer.type=sync
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder

`

server2(IP:192.168.50.142):

kafka/config/zookeeper.properties `

dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0

`

kafka/config/server.properties `

broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
broker.id=0
port=9092
log.dir=/tmp/kafka-logs-1
delete.topic.enable=true

`

kafka.config/consumer.properties `

zookeeper.connect=127.0.0.1:2181
zookeeper.connection.timeout.ms=6000
group.id=test-consumer-group

`

kafka_2.11-0.9.0.0 java 1.8.0_60 node v4.4.4

Should I need to change any configuration? Please give some help.

like image 752
Eddy Kim Avatar asked May 25 '16 04:05

Eddy Kim


People also ask

What is cluster in Kafka?

Kafka cluster typically consists of multiple brokers to maintain load balance. Kafka brokers are stateless, so they use ZooKeeper for maintaining their cluster state.

How many nodes are in Kafka cluster?

5 Nodes (recommended): A five-node cluster allows you to take one server out for maintenance or upgrade and still be able to take a second unexpected failure, without interrupting your service. 7 Nodes (recommended): The same as for 5-node cluster but with the ability to bear the failure of three nodes.

What is Kafka configuration?

It dictates how the Kafka tool runs in the JAAS configuration. These are some security rules and regulations used while exchanging words with the servers. It indicates the size of the memory buffer which will hold the data to be sent to the producer.


2 Answers

It seems your producer configurations are not correct.

kafka/config/producer.properties

bootstrap.servers=192.168.50.142:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
like image 53
Kamal Chandraprakash Avatar answered Sep 22 '22 06:09

Kamal Chandraprakash


Make the following changes in server.properties:

  1. Change the second line to

    listeners=PLAINTEXT://:9092
    
  2. Add this line:

    advertised.listeners=PLAINTEXT://192.168.50.142:9092
    

    This line is needed because this is the hostname and port the broker will advertise to consumers and producers. And since you have a producer on another machine, this line is needed.

While writing any command in the terminal, use

    <command> --zookeeper localhost:2181 <rest of it>

Hope this works.

like image 43
Chandan Avatar answered Sep 21 '22 06:09

Chandan