Monitoring Kafka Clusters , Producers and Consumers

Question

I am new to Kafka and I have three nodes Kafka cluster with multiple consumers and producers.I want to monitor the Kafka cluster on the basis of three parameters :

1.Topic wise monitoring - to get incoming rate of topic , lag in a topic, number of active consumers attached with the topic, number of partitions
2.Consumer based monitoring - the topic from which it is consuming,consumption speed. a number of messages consumed.
3.Message-Based Monitoring - topic name in which message is produced,and status of message i.e whether it's processed or pending.

I have tried various monitoring tools like (KafkaOffsetMonitor,Burrow) but none of them are matching my use case .So can anyone suggest how to setup this kind of monitoring system?.Are there APIs available through which I can get the above-mentioned details

Answer 1

you could fire up JConsole and monitor the whole Kafka cluster with JMX configured. See https://kafka.apache.org/documentation#monitoring for details.