kafka vs chronicle queue vs disruptor

Question

Anyone can compare the underlying design and performance among kafka, chronicle queue and disruptor, in terms of logging? Seems kafka has most users but don't avoid GC.

Answer 1

I think you might be confused about how Kafka is used in logging pipelines - typically it's for "shipping logs" from a single process (local disk) to a log database like Elasticsearch or Splunk and the performance is going to be on the order of 100K messages/s for a single machine e.g. see https://www.confluent.io/blog/kafka-fastest-messaging-system/. The reason you might use Kafka is to "protect" your database from bursts e.g. https://logz.io/blog/deploying-kafka-with-elk/.

Chronicle Queue and Disrupter would be for simply writing the logs to the local disk and can achieve on the order of 10M lines/s e.g. see https://grobmeier.solutions/log4j-2-performance-close-to-insane-20072013.html#.Ue02Z2RATzc.

You might further wonder what's the point of writing 10M lines/s to disk if you can only ship 100K/s.

The reason is that you probably only write 10M lines/s when something bad is happening (or you're debugging) and only for a short period of time so if you're reading from disk into Kafka then your "reader" will get behind but can eventually catch up once the burst is over.