Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Difference in kafka 0.8 with kafka 0.10

Tags:

apache-kafka

kafka-consumer-api

kafka-producer-api

Can anybody list out basic differentiating features in Kafka 0.10 as compared to kafka 0.8.

We are thinking of upgrading our kafka-8 to kafka-10. What challenges can one face in upgrading?

Thanks in advance.

like image 782
Nilotpal Avatar asked Jun 08 '16 06:06

Nilotpal

People also ask

What are different types of Kafka?

Kafka supports two types of topics: Regular and compacted. Regular topics can be configured with a retention time or a space bound. If there are records that are older than the specified retention time or if the space bound is exceeded for a partition, Kafka is allowed to delete old data to free storage space.

What is the current Kafka version?

3.2. 1 is the latest release. The current stable version is 3.2.

What is the max message size in Kafka?

The Kafka max message size is 1MB. In this lesson we will look at two approaches for handling larger messages in Kafka. Kafka has a default limit of 1MB per message in the topic. This is because very large messages are considered inefficient and an anti-pattern in Apache Kafka.

What is the difference between Apache Kafka and Kafka streams?

Apache Kafka is the most popular open-source distributed and fault-tolerant stream processing system. Kafka Consumer provides the basic functionalities to handle messages. Kafka Streams also provides real-time stream processing on top of the Kafka Consumer client.

4 Answers

0.10 adds the following over 0.8

  • Kafka Streams API
  • Kafka Connect API
  • new unified Consumer API
  • transport encryption using TLS/SSL
  • Kerberos/SASL Authentication support
  • Access Control Lists
  • timestamps on messages
  • reduced client dependence on zookeeper (offsets stored in Kafka topic)
  • client interceptors
  • lots and lots of bug fixes and improvements
like image 102
Hans Jespersen Avatar answered Oct 12 '22 20:10

Hans Jespersen

You can simply identify the upgrade changes and effects if you're willing to upgrade it to 0.10.

Upgrading from 0.8.x to 0.10.0.0

like image 39
Kulasangar Avatar answered Oct 12 '22 19:10

Kulasangar

This major issue has been fixed in 0.10:

The memory and bandwidth utilization of 0.9 clients is very high. Such issues were causing spike in resource utilization by the kafka brokers. P.S.: https://github.com/elodina/go_kafka_client/issues/52

like image 36
ininprsr Avatar answered Oct 12 '22 20:10

ininprsr

Kafka 0.10 changes

  • Zookeeper connections are discouraged . From 0.10 there won't be any zookeeper connections required . All connections for consuming data will be maintained by consumer API
  • zookeeper consumer config path will not have any offset lag data
  • Kafka streams API and connect API has been introduced
  • in Kafka 0.11 transactions have been introduced which enables Kafka to have exactly once delivery
  • for monitoring offsets there is a new class KafkaConsumerGroup command class
like image 30
Sunjay Jeffrish Avatar answered Oct 12 '22 21:10

Sunjay Jeffrish
Sign in to Comment

Related questions

How to implement FlinkKafkaProducer serializer for Kafka 2.2
Why swapping is not a good idea in zookeeper and kafka?
Why we require Apache Kafka with NoSQL databases?
Kafka - Could not find a 'KafkaClient' entry in the JAAS configuration java
Kafka: How to Display Offsets
Pyspark 2.4.0, read avro from kafka with read stream - Python
Kafka Offset and Partition identification
java Kafka producer error
Kafka Memory requirement
Can anyone share a Flink Kafka example in Scala?
How can I get the group.id of a topic in command line in Kafka?
Kafka Streams: how to write to a topic?
Put() vs Flush() in Kafka Connector Sink Task
Separate zookeeper install or not using kafka 10.2?
push multiple Line Text as one message in a kafka topic
Unknown magic byte with kafka-avro-console-consumer
How to create kafka compacted topic
Cleanup Policy:Compact/Delete and log.retention
Docker compose doesn't save data in volume
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!