Kafka instead of Rest for communication between microservices

Question

I want to change the communication between (micro)-services from REST to Kafka. I'm not sure about the topics and wanted to hear some opinions about that.

Consider the following setup: I have an API-Gateway that provides CRUD functions via REST for web applications. So I have 4 endpoints which users can call. The API-Gateway will produce the request and consumes the responses from the second service. The second service consumes the requests, access the database to execute the CRUD operations on the database and produces the result.

How many topics should I create? Do I have to create 8 (2 per endpoint (request/response)) or is there a better way to do it?

Would like to hear some experience or links to talks / documentation on that.

Answer 1

The short answer for this question is; It depends on your design.

You can use only one topic for all your operations or you can use several topics for different operations. However you must know that;

Your have to produce messages to kafka in the order that they created and you must consume the messages in the same order to provide consistency. Messages that are send to kafka are ordered within a topic partition. Messages in different topic partitions are not ordered by kafka. Lets say, you created an item then deleted that item. If you try to consume the message related to delete operation before the message related to create operation you get error. In this scenario, you must send these two messages to same topic partition to ensure that the delete message is consumed after create message.

Please note that, there is always a trade of between consistency and throughput. In this scenario, if you use a single topic partition and send all your messages to the same topic partition you will provide consistency but you cannot consume messages fast. Because you will get messages from the same topic partition one by one and you will get next message when the previous message consumed. To increase throughput here, you can use multiple topics or you can divide the topic into partitions. For both of these solutions you must implement some logic on producer side to provide consistency. You must send related messages to same topic partition. For instance, you can partition the topic into the number of different entity types and you send the messages of same entity type crud operation to the same partition. I don't know whether it ensures consistency in your scenario or not but this can be an alternative. You should find the logic which provides consistency with multiple topics or topic partitions. It depends on your case. If you can find the logic, you provide both consistency and throughput.

For your case, i would use a single topic with multiple partitions and on producer side i would send related messages to the same topic partition.

--regards