Kafka : Use common consumer group to access multiple topics

Question

Our cluster runs Kafka 0.11 and has strict restrictions on using consumer groups. We cannot use arbitrary consumer groups so Admin has to create required consumer groups.

We run Kafka Connect HDFS Sinks to read data from topics and write to HDFS. All the topics have only one partition.

I can consider following two patterns when using Consumer Groups in Kafka HDFS Sink.

As shown in the pictures:

Case 1: Each topic has its own Consumer Group

Case 2: All the topics have a common Consumer Group

I am aware that when a topic has multiple partitions, and if a consumer failed, another consumer in the same consumer group take over that partition.

My question :

Does the same thing happen when multiple topics share the same consumer group? ie: if a Consumer failed(HDFS Sink), will another Consumer(HDFS Sink connector) takeover the work and read from that topic?

Update: Each Kafka HDFS Sink Connector subscribed to only one topic.

Answer 1

I'm surprised that all answers with "yes" are wrong. I just tested it and having the same group.id for consumers for different topic works well and does NOT mean that they share messages, because for Kafka the key is (topic, group) rather than just (group). Here is what I did:

1. created 2 different topics T1 and T2 with 2 partitions in each topic
2. created 2 consumers with the same group xxx
3. assigned consumer C1 to T1, consumer C2 to T2
4. produced messages to T1 - only consumer C1 assigned to T1 processed them
5. produced messages to T2 - only consumer C2 assigned to T2 processed them
6. killed consumer C1 and repeated 4-5 steps. Only consumer C2 processed messages from T2
7. messages from T1 were not processed

Conclusion: Consumers with the same group name subscribed to different topics will NOT consume messages from other topics, because the key is (topic, group)