Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka how to consume one topic parallel

Tags:

apache-kafka

I read kafka document, still don't know how consume one topic parallel?

Suppose: I have one topic like "something happened" (don't split this topic), and I have many customers that want to consume it. So what should I do, so that multiple customers can consume it parallel? Should I use partitioning and customer groups?

I have one idea about this, but I'm not sure whether is it right.

Make many partitions about the same topic, and make one partition to one customer, so one producer must produce the same to these partitions, and every customer in the different customer group, is it right?

like image 453
regrecall Avatar asked Sep 01 '14 09:09

regrecall


1 Answers

Using partitions is the way of being able to parallelize the consumption of a topic. Let´s say you have 10 partitions for your topic, then you can have 10 consumers in the same consumer group reading one partition each. If you have less consumers than partitions, then they would be responsible for more than one partition each. If you have more consumers than partitions, then there would be consumers who would not get any partition assigned to them and have nothing to do except being available to replace another consumer who has died.

like image 66
Lundahl Avatar answered Oct 04 '22 01:10

Lundahl