Communication among microservices: Apache Kafka vs Hazelcast's Topic

Question

Disclaimer. I have experience with Hazelcast and Vert.x. I'm new to Apache Kafka. Sorry if my question look preconceived, it's not.

There are two widespread ways to arrange communication among microservices: REST and messaging. In my region, when someone says they're using messaging for communication among microservices - it de facto means Apache Kafka.

I am interested to know why Apache Kafka is better fit for communication needs among microservices than Hazelcast's Topic? Is it better? Because of which guarantees, features or architecture decisions?

The Hazelcast's example for cluster wide messaging looks as following:

// node #1
Hazelcast.newHazelcastInstance()
         .getTopic("topic")
         .publish(new Date());

// node #2
Hazelcast.newHazelcastInstance()
         .getTopic("topic");
         .addMessageListener(message -> /*Do something here*/);

Also there is Vert.x (very roughtly speaking actors framework) written on top of Hazelcast's topics and member discovery.

Is Kafka messaging better for communication among microservices?

Answer 1

This is a bit generic question, and I'm not a Kafka expert; but I will try to tell about Hazelcast's messaging features.

Hazelcast contains two types of topics; one is regular ITopic and the other one is Reliable Topic, which again implements ITopic interface. The usages are mostly the same, but they differ in what they guarantee. Regular ITopic is based on Hazelcast's eventing mechanism, and does not guarantee message delivery. Reliable Topic is backed up by the Ringbuffer, and the events are not lost since the Ringbuffer is configured with one synchronous backup by default. Also, each Reliable ITopic gets its own Ringbuffer; if a topic has a very fast producer, it will not lead to problems at topics that run at a slower pace. Lastly, since the event system behind a regular ITopic is shared with other data structures, e.g., collection listeners, you can run into isolation problems. This does not happen with the Reliable ITopic. But with all these disadvantages, regular ITopic can run a bit faster since it uses fire&forget eventing mechanism.

Apache Kafka has its own great advantages; such as it is all built as a message streaming platform with a temporal persistent log, and therefore has ability to store data to disk for fault-tolerance.

In summary, if you need message persistence with all features needed for messaging on your application, go with Kafka since it is more specialized. But if you need a in-memory data platform including support for messaging, go with Hazelcast.