Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

kafka-console-consumer custom deserializer

I would like to use my custom value.deserializer when using the kafka-console-consumer command line tool. Something like this:

./kafka-console-consumer --bootstrap-server kafka2:29092 \
                         --property value.deserializer=My.Custom.KafkaDeserializer \
                         --topic TEST

But its unable to find my custom class...

Exception in thread "main" java.lang.ClassNotFoundException: My.Custom.KafkaDeserializer

How can I reference the appropriate jar file so that the script will recognize it?

like image 575
Kevin Noreus Avatar asked Aug 09 '17 03:08

Kevin Noreus


People also ask

What is Kafka deserializer?

Serialization is the process of converting objects into bytes. Deserialization is the inverse process — converting a stream of bytes into an object. In a nutshell, it transforms the content into readable and interpretable information.

How do you deserialize Kafka messages?

How do I get the java application to appropriately deserialize a kafka message into a POJO? To reproduce the issue the following things have to be done: Run the command docker-compose up -d to bring up 3 instances of Zookeeper and Kafka. It also brings up Kafdrop that connects to the Kafka cluster.


2 Answers

You need to place the deserializer jar under the "libs" folder of your Kafka server.

like image 173
Nagarjuna Nag Avatar answered Nov 11 '22 15:11

Nagarjuna Nag


As already said there is an opened JIRA for that (https://issues.apache.org/jira/browse/KAFKA-2526) and a custom deserializer can't be used yet.

like image 39
ppatierno Avatar answered Nov 11 '22 17:11

ppatierno