Kafka-python producer enable.idempotent

Question

i'm facing the duplication data in kafka so i want to use idempotent in kafka-python but unfortunately its not the part of it, how i can stop the duplication data in same partition with out idempotent or there is any way to import it or customize the current library? Thanks in advance.

Answer 1

You'd better use confluent_kafka, since kafka-python does not provide this feature. Have a look at this link

producer = Producer({'bootstrap.servers': ‘localhost:9092’,
          'enable.idempotence': True})

This feature was made available with the 1.0.0 release of librdkafka (a library written in C).