Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Azure Event Hub - multiple event types in same Event Hub

I have set up Azure Event Hub. There are 2 publishers:

  1. Publisher 1 (with Send Policy)

  2. Publisher 2 (with Send Policy)

Publisher 1 will send Event 1 and Publisher 2 will send Event 2. Both Event 1 & Event 2 are different format.

Question 1: This would mean we have different messages in EH - what are the tradeoff with this approach? Should I instead create 2 EH (one for Publisher 1 and another for Publisher 2)? What is the best practice and design philosophy?

If I go with above approach, I would have to set up a Consumer with Listen Policy to look for these events and parse/transform these events and de-serialize these.

Question 2: Would I need 2 Consumers (Consumer 1 and Consumer 2) to read messages meant for them ? (Consumer 1 will read only Event 1 and Consumer 2 will read only Event 2)?

like image 301
khar Avatar asked Jan 30 '23 13:01

khar


1 Answers

Scenario 1: One Event Hub for multiple event types

In this scenario you have multiple options when it comes to sending and processing the messages:

  1. Just send messages to the Event Hub. Write a consuming process that reads the messages and based on the type handle them different.
  2. Just send messages to the Event Hub. Create different consumer groups, one for each message type. Have 2 processes that both read their own consumer group and skip the messages that they are not qualified to handle. (Each consumer group essentially read the same data, but they can be at different positions in the data stream). See Azure event hubs and multiple consumer groups
  3. Send the messages to a designated partition. For example, have messages of type A send to partition 1 and messages of type B to partition 2. This can however affect the scalability of the event hub. Create dedicated processes per (set of) partition(s). Each process will then only process messages of the same type. See https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-programming-guide#partition-key and https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-programming-guide#create-a-partition-sender

Scenario 2: An Event Hub per event types

This one is easy, just create 2 event hubs and 2 consuming processes. But you'll have to manage 2 event hubs and given the abilities of an Event Hub it might be overkill.

My advice

It depends on the amount of data, but based on my experience I would go to send all messages the one event hub and have one process reading the message and perform an action based on the message type using some c# code.

like image 80
Peter Bons Avatar answered Feb 02 '23 09:02

Peter Bons