python vs java for kafka implementation

Question

@kafka users:

I have been trying to understand python client for kafka, including PyPy client as well. Following was a good benchmarking i read and realized some similar results:

http://mrafayaleem.com/2016/03/31/apache-kafka-producer-benchmarks/

I am extremely confused whether Java has a massive advantage over python, as the libraries are written using Java and kafka. So my question is, does the native implementation of Kafka in Java helps with performance tremendously when Java is used, or PyPy/Python works equally better?

Being a python programmer, I am not at all comfortable with java, and hence confused.

Answer 1

Apache Kafka defines a language neutral wire protocol (see https://kafka.apache.org/protocol) and so clients can be written in any programming language and do not need to be based on the Java client that ships with the core Kafka distribution. For example, the c/c++ librdkafka library is a very high performance non-Java client. There are a number of python Kafka clients including one that is based on librdkafka. Benchmark results and other comparison information for the various Kafka Python clients is available here http://activisiongamescience.github.io/2016/06/15/Kafka-Client-Benchmarking/