Kafka independent consumers

Question

I am new to Kafka and trying to validate my design. Below is what I have.

I have one producer publishing to one topic and have a bunch of containers (where my web app is deployed) where a consumer is running on each one of the containers. These consumers are not in the consumer-group and consuming messages independently. Each consumer should read all messages from the topic. e.g. suppose there are 3 messages on the topic - m0, m1, m2 then consumer1 to consumerN should read m0, m1, m2 independently. Each consumer is committing offset as soon as it processes the read messages. Whenever there is a re-deployment of my web application, all consumers go down and are restarted. They are supposed to start reading from their last committed offset.

For this set-up, as I said all my consumers are not in any consumergroup and I suppose this should make them read messages independently, please confirm that is right? Is there any flaw in this design?

Answer 1

your design is just missing a small thing, each of your container must define a different (kafka) consumer group and reuse it at restart. Otherwise a random group id will be assigned to each consumer at each startup and therefore they will not pick up consuming where they left at shutdown.