Which Language to use for Kafka Consumer

Question

I am trying to set up a Kafka system. Since most of the existing code in my project is already in PHP, I will most probably be writing the producers in PHP itself. But I am comparatively very less constrained when it comes to choosing a language to write the consumer. Now, that there are so many clients which can be used I am in a fix.

In other to order to choose the right tech here, what are the various factors that should be kept in mind?

Would especially like to apply this knowledge to choose between java client vs node client(multithreaded model vs async model)

Any help will be highly appreciated.

Answer 1

The Java client is the most advance client and officially supported by the Kafka Project -- most other clients are third party projects and many do not implement all available features.

Thus, I would recommend to use Java clients.

Answer 2

Kafka is basically written in pure Java and Kafka’s native API is java, so this is the only language where you’re not using a third-party library.You always have an edge over writing in other languages which have an additional overhead.

Node.js isn’t optimized for high throughput applications such as Kafka. So if you need the high processing rates that come standard on Kafka, or perhaps C++.

Also, I believe Kafka consumer clients written in Java has good community support. So it makes sense to implement it using java as long as you don't have any other dependency stopping you from implementing it from.

Also, check this out for the benchmarking results using various Kafka Clients. The results are contrasting.

Client Type Throughput(No of messages)
Java        40,000 - 50,0000
Go          28,000 - 30,0000
Node         6,000 -  8,0000
Kafka-pixy     700 -    800
Logstash       250