What is Consumer Group in azure Event Hub?

Question

Can anyone please figure out what is Consumer Group in Azure Event Hubs. And What is the use of it? I have surfed a lot of sites but I can't get a clear answer.

Answer 1

From the docs:

Consumer groups: A view (state, position, or offset) of an entire event hub. Consumer groups enable consuming applications to each have a separate view of the event stream. They read the stream independently at their own pace and with their own offsets.

Diagram:

According to this consumer groups are the way to logically separate your consumers, so they only see the events they are interested in.

Answer 2

Think of a consumer group as a label that groups one or more event consumers together as a set. It's often named after what responsibility the consumer has in an application (ex: "Telemetry", "OrderProcessing"). A default consumer group named "$Default" is created when an Event Hub is created.

Like mentioned in the previous answer, consumer groups allow multiple applications to have their own view of an event stream, thus allowing each to read the stream independently and at their own speed. For example, you might have a downstream storage writer application that writes event data to long-term storage, and then another application that performs complex event processing--the two applications would belong to two different consumer groups.

Since an Event Hubs consumer is associated with a specific Event Hubs and consumer group, if you pass in the same consumer group as a parameter when constructing EventHubConsumerClients, then those clients will be associated with the same consumer group (thus grouping the event consumers):

var consumer = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);

Note that you can only have multiple consumer groups if you're using the Standard tier service.

Answer 3

In Azure Event Hub, consumer groups are a means of limiting the number of concurrent readers to the same partitions.

They serve no real purpose except perhaps a practical one which is to guide you away from reading the same events multiple times from the same system or application.

There's also a pricing aspect which is that to enable multiple applications subscribing to the same event stream, you may need to upgrade to a better plan (for example, the basic plan allows just a single consumer group).

As others have pointed out, the documentation explains consumer groups as a "view of an entire event hub". This view, however, has no practical bearing.

You could run 5 different applications on a single consumer group, but it is impractical perhaps, because this won't scale to the 6th application should you ever need it since there is a limit of 5 concurrent readers to the same partition per consumer group.

Comparison to Kafka consumer groups

Event Hub consumer groups are entirely different than Kafka consumer groups (whereby offsets are stored automatically as you progress on reading from the topic).

Clarification here: https://github.com/Azure/azure-event-hubs-for-kafka#more-faq.

When using the Kafka API, Event Hubs do support Kafka consumer groups.