Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Apache Spark: Getting a InstanceAlreadyExistsException when running the Kafka producer

I have an small app in scala that creates kafka producer and that run with Apache Spark. when I run the command

spark-submit --master local[2] --deploy-mode client <into the jar file> <app Name> <kafka broker> <kafka in queue> <kafka out queue> <interval>

I am getting this WARN: WARN AppInfoParser: Error registering AppInfo mbean javax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=

The code is not relevant because I am getting this exception when scala creates the KafkaProducer: val producer = new KafkaProducerObject,Object

Does anybody have a solution for this? Thank you!

like image 273
CMPE Avatar asked Jan 05 '23 19:01

CMPE


1 Answers

When a Kafka Producer is created, it attempts to register an MBean using the client.id as its unique identifier.

There are two possibilities of why you are getting the InstanceAlreadyExistsException warning:

  1. You are attempting to initialize more than one Producer at a time with the same client.id property on the same JVM.
  2. You are not calling close() on an existing Producer before initializing another Producer. Calling close() unregisters the MBean.

If you leave the client.id property blank when initializing the producer, a unique one will be created for you. Giving your producers unique client.id values or allowing them to be auto-generated would resolve this problem.

In the case of Kafka, MBeans can be used for tracking statistics. MBeans created by Kafka displayed in JMC.exe

like image 110
Steve Avatar answered Jan 15 '23 21:01

Steve